Compare commits

...

27 Commits

Author SHA1 Message Date
Tyrrrz 3aa90bb27b Update version 2020-12-29 21:02:46 +02:00
Tyrrrz 51acffc92e [HTML] Fix up replies 2020-12-29 20:49:34 +02:00
Tyrrrz 39a33538fe Cleanup last PR 2020-12-28 22:10:46 +02:00
Sanqui 07ac805677 Add support for replies (#455) 2020-12-28 22:03:30 +02:00
Alexey Golub 097a75e326 Update readme 2020-12-28 18:38:22 +02:00
Tyrrrz 3d9ee3b339 Embrace Snowflake as first class citizen 2020-12-27 19:42:21 +02:00
multisignature 4ff7990967 Fix grammar in readme (#466) 2020-12-23 21:00:33 +02:00
Andrew Kolos 730fed075f [CLI] Allow passing after and before message ids instead of dates (#462) 2020-12-22 17:41:40 +02:00
Alexey Golub 600c354ab2 Use Discord API v8 2020-12-19 18:37:38 +02:00
Alexey Golub d8dc12b3f7 Update readme 2020-12-19 18:22:03 +02:00
Alexey Golub 33a6d49d76 Use .NET 5.0 SDK in Docker 2020-12-19 18:11:31 +02:00
Tyrrrz 3861dba60c Refactor last PR 2020-12-17 17:49:35 +02:00
Andrew Kolos 69acb5bf76 Set last write time on downloaded media to the value provided in the Last-Modified header (#459) 2020-12-17 17:13:47 +02:00
Tyrrrz 0e67dcbf2e Update readme 2020-12-15 17:47:00 +02:00
Tyrrrz 45d2be549d Install dotnet 5 on CI/CD so that source generators work 2020-12-15 16:17:37 +02:00
Tyrrrz 9394824943 Update MiniRazor to use build-time template compilation 2020-12-15 03:44:13 +02:00
Sanqui 43c50324d8 [JSON] Don't escape Unicode characters when possible (#453)
While UnsafeRelaxedJsonEscaping sounds scary, there is no risk
here since the JSON data is saved directly into a file.  See #450
2020-12-14 21:57:48 +02:00
Alexey Golub a6353de2a7 Use preview language version 2020-12-14 18:48:34 +02:00
Sanqui d020652f50 [CLI] Display channel name for errors during ExportMultipleAsync (#452) 2020-12-14 18:48:11 +02:00
Tyrrrz 63803f98aa Use C#9 features 2020-12-14 17:45:42 +02:00
Tyrrrz 9dda9cfc27 Use floating dotnet versions on CI/CD 2020-12-14 17:37:49 +02:00
Tyrrrz 61d0ddeced Update project structure 2020-12-10 16:52:05 +02:00
Tyrrrz 4c64977b71 Expect null result in WaitAndRetry logic in http retry handler
Fixes #447
2020-12-07 16:24:33 +02:00
Tyrrrz 8bc5b547ef Remove gitattributes and simplify gitignore 2020-12-07 16:23:53 +02:00
Lucas LaBuff 286981d9f8 [HTML] Prettier attachment preview for non-media files (#448) 2020-12-07 16:20:38 +02:00
Tyrrrz 5a62ce6af9 Update JsonExtensions to 1.0.1 2020-11-29 00:42:05 +02:00
Tyrrrz d72fe594b4 Use JsonExtensions package 2020-11-29 00:17:58 +02:00
76 changed files with 804 additions and 774 deletions
-63
View File
@@ -1,63 +0,0 @@
###############################################################################
# Set default behavior to automatically normalize line endings.
###############################################################################
* text=auto
###############################################################################
# Set default behavior for command prompt diff.
#
# This is need for earlier builds of msysgit that does not have it on by
# default for csharp files.
# Note: This is only used by command line
###############################################################################
#*.cs diff=csharp
###############################################################################
# Set the merge driver for project and solution files
#
# Merging from the command prompt will add diff markers to the files if there
# are conflicts (Merging from VS is not affected by the settings below, in VS
# the diff markers are never inserted). Diff markers may cause the following
# file extensions to fail to load in VS. An alternative would be to treat
# these files as binary and thus will always conflict and require user
# intervention with every merge. To do so, just uncomment the entries below
###############################################################################
#*.sln merge=binary
#*.csproj merge=binary
#*.vbproj merge=binary
#*.vcxproj merge=binary
#*.vcproj merge=binary
#*.dbproj merge=binary
#*.fsproj merge=binary
#*.lsproj merge=binary
#*.wixproj merge=binary
#*.modelproj merge=binary
#*.sqlproj merge=binary
#*.wwaproj merge=binary
###############################################################################
# behavior for image files
#
# image files are treated as binary by default.
###############################################################################
#*.jpg binary
#*.png binary
#*.gif binary
###############################################################################
# diff behavior for common document formats
#
# Convert binary document formats to text before diffing them. This feature
# is only available from the command line. Turn it on by uncommenting the
# entries below.
###############################################################################
#*.doc diff=astextplain
#*.DOC diff=astextplain
#*.docx diff=astextplain
#*.DOCX diff=astextplain
#*.dot diff=astextplain
#*.DOT diff=astextplain
#*.pdf diff=astextplain
#*.PDF diff=astextplain
#*.rtf diff=astextplain
#*.RTF diff=astextplain
+7 -2
View File
@@ -13,10 +13,15 @@ jobs:
- name: Checkout
uses: actions/checkout@v2.3.3
- name: Install .NET
- name: Install .NET (v3.1)
uses: actions/setup-dotnet@v1.7.2
with:
dotnet-version: 3.1.100
dotnet-version: 3.1.x
- name: Install .NET (v5.0)
uses: actions/setup-dotnet@v1.7.2
with:
dotnet-version: 5.0.x
- name: Build & publish (CLI)
run: dotnet publish DiscordChatExporter.Cli/ -o DiscordChatExporter.Cli/bin/Publish/ --configuration Release
+7 -2
View File
@@ -10,10 +10,15 @@ jobs:
- name: Checkout
uses: actions/checkout@v2.3.3
- name: Install .NET
- name: Install .NET (v3.1)
uses: actions/setup-dotnet@v1.7.2
with:
dotnet-version: 3.1.100
dotnet-version: 3.1.x
- name: Install .NET (v5.0)
uses: actions/setup-dotnet@v1.7.2
with:
dotnet-version: 5.0.x
- name: Build & publish (CLI)
run: dotnet publish DiscordChatExporter.Cli/ -o DiscordChatExporter.Cli/bin/Publish/ --configuration Release
+6 -246
View File
@@ -1,261 +1,21 @@
## Ignore Visual Studio temporary files, build results, and
## files generated by popular Visual Studio add-ons.
# User-specific files
*.suo
*.user
*.userosscache
*.sln.docstates
# User-specific files (MonoDevelop/Xamarin Studio)
*.userprefs
.idea/
# Build results
[Dd]ebug/
[Dd]ebugPublic/
[Rr]elease/
[Rr]eleases/
x64/
x86/
[Xx]64/
[Xx]86/
[Bb]uild/
bld/
[Bb]in/
[Oo]bj/
[Ll]og/
# Visual Studio 2015 cache/options directory
.vs/
# Uncomment if you have tasks that create the project's static files in wwwroot
#wwwroot/
# MSTest test Results
[Tt]est[Rr]esult*/
[Bb]uild[Ll]og.*
# NUNIT
*.VisualState.xml
TestResult.xml
# Build Results of an ATL Project
[Dd]ebugPS/
[Rr]eleasePS/
dlldata.c
# DNX
project.lock.json
project.fragment.lock.json
artifacts/
*_i.c
*_p.c
*_i.h
*.ilk
*.meta
*.obj
*.pch
*.pdb
*.pgc
*.pgd
*.rsp
*.sbr
*.tlb
*.tli
*.tlh
*.tmp
*.tmp_proj
*.log
*.vspscc
*.vssscc
.builds
*.pidb
*.svclog
*.scc
# Chutzpah Test files
_Chutzpah*
# Visual C++ cache files
ipch/
*.aps
*.ncb
*.opendb
*.opensdf
*.sdf
*.cachefile
*.VC.db
*.VC.VC.opendb
# Visual Studio profiler
*.psess
*.vsp
*.vspx
*.sap
# TFS 2012 Local Workspace
$tf/
# Guidance Automation Toolkit
*.gpState
# ReSharper is a .NET coding add-in
_ReSharper*/
*.[Rr]e[Ss]harper
*.DotSettings.user
# JustCode is a .NET coding add-in
.JustCode
# TeamCity is a build add-in
_TeamCity*
# DotCover is a Code Coverage Tool
*.dotCover
# NCrunch
_NCrunch_*
.*crunch*.local.xml
nCrunchTemp_*
# MightyMoose
*.mm.*
AutoTest.Net/
# Web workbench (sass)
.sass-cache/
# Installshield output folder
[Ee]xpress/
# DocProject is a documentation generator add-in
DocProject/buildhelp/
DocProject/Help/*.HxT
DocProject/Help/*.HxC
DocProject/Help/*.hhc
DocProject/Help/*.hhk
DocProject/Help/*.hhp
DocProject/Help/Html2
DocProject/Help/html
# Click-Once directory
publish/
# Publish Web Output
*.[Pp]ublish.xml
*.azurePubxml
# TODO: Comment the next line if you want to checkin your web deploy settings
# but database connection strings (with potential passwords) will be unencrypted
#*.pubxml
*.publishproj
# Microsoft Azure Web App publish settings. Comment the next line if you want to
# checkin your Azure Web App publish settings, but sensitive information contained
# in these scripts will be unencrypted
PublishScripts/
# NuGet Packages
*.nupkg
# The packages folder can be ignored because of Package Restore
**/packages/*
# except build/, which is used as an MSBuild target.
!**/packages/build/
# Uncomment if necessary however generally it will be regenerated when needed
#!**/packages/repositories.config
# NuGet v3's project.json files produces more ignoreable files
*.nuget.props
*.nuget.targets
# Microsoft Azure Build Output
csx/
*.build.csdef
# Microsoft Azure Emulator
ecf/
rcf/
# Windows Store app package directories and files
AppPackages/
BundleArtifacts/
Package.StoreAssociation.xml
_pkginfo.txt
# Visual Studio cache files
# files ending in .cache can be ignored
*.[Cc]ache
# but keep track of directories ending in .cache
!*.[Cc]ache/
# Others
ClientBin/
~$*
*~
*.dbmdl
*.dbproj.schemaview
*.jfm
*.pfx
*.publishsettings
node_modules/
orleans.codegen.cs
# Since there are multiple workflows, uncomment next line to ignore bower_components
# (https://github.com/github/gitignore/pull/1529#issuecomment-104372622)
#bower_components/
# RIA/Silverlight projects
Generated_Code/
# Backup & report files from converting an old project file
# to a newer Visual Studio version. Backup files are not needed,
# because we have git ;-)
_UpgradeReport_Files/
Backup*/
UpgradeLog*.XML
UpgradeLog*.htm
# SQL Server files
*.mdf
*.ldf
# Business Intelligence projects
*.rdl.data
*.bim.layout
*.bim_*.settings
# Microsoft Fakes
FakesAssemblies/
# GhostDoc plugin setting file
*.GhostDoc.xml
# Node.js Tools for Visual Studio
.ntvs_analysis.dat
# Visual Studio 6 build log
*.plg
# Visual Studio 6 workspace options file
*.opt
# Visual Studio LightSwitch build output
**/*.HTMLClient/GeneratedArtifacts
**/*.DesktopClient/GeneratedArtifacts
**/*.DesktopClient/ModelManifest.xml
**/*.Server/GeneratedArtifacts
**/*.Server/ModelManifest.xml
_Pvt_Extensions
# Paket dependency manager
.paket/paket.exe
paket-files/
# FAKE - F# Make
.fake/
# JetBrains Rider
.idea/
*.sln.iml
# CodeRush
.cr/
# Python Tools for Visual Studio (PTVS)
__pycache__/
*.pyc
# Coverage
*.opencover.xml
+13
View File
@@ -1,3 +1,16 @@
### v2.26 (29-Dec-2020)
- [HTML] Added support for replies. (Thanks [@Sanqui](https://github.com/Sanqui))
- [HTML] Improved styling of non-media attachments (regular files, i.e. not video, audio, or image). (Thanks [@Lucas LaBuff](https://github.com/96-LB))
- [HTML] Improved initial export performance.
- [JSON] Relaxed escaping rules for unicode characters. (Thanks [@Sanqui](https://github.com/Sanqui))
- [JSON] Added support for replies. (Thanks [@Sanqui](https://github.com/Sanqui))
- [CLI] Added channel names to errors shown on failures when exporting multiple channels. (Thanks [@Sanqui](https://github.com/Sanqui))
- [CLI] Added support for passing message IDs (snowflakes) to `--after` and `--before` options. (Thanks [@Andrew Kolos](https://github.com/andrewkolos))
- Files downloaded by the "download referenced media" option now have their dates set according to the `Last-Modified` header provided by Discord CDN. (Thanks [@Andrew Kolos](https://github.com/andrewkolos))
- Fixed an issue where the application sometimes crashed with `NullReferenceException` when retrying a failed HTTP request.
- Migrated to Discord API v8.
### v2.25 (23-Nov-2020)
- [JSON] Added `mentions` property to the output. This contains users referenced inside the message text, but doesn't include mentions in embeds. (Thanks [@Andrew Kolos](https://github.com/andrewkolos))
@@ -2,12 +2,12 @@
<PropertyGroup>
<TargetFramework>netcoreapp3.1</TargetFramework>
<Version>2.25</Version>
<Version>2.26</Version>
<Company>Tyrrrz</Company>
<Copyright>Copyright (c) Alexey Golub</Copyright>
<LangVersion>latest</LangVersion>
<LangVersion>preview</LangVersion>
<Nullable>enable</Nullable>
<WarningsAsErrors>nullable</WarningsAsErrors>
</PropertyGroup>
</Project>
</Project>
@@ -1,10 +1,10 @@
using System;
using System.IO;
using System.IO;
using System.Threading.Tasks;
using CliFx;
using CliFx.Attributes;
using CliFx.Exceptions;
using CliFx.Utilities;
using DiscordChatExporter.Domain.Discord;
using DiscordChatExporter.Domain.Discord.Models;
using DiscordChatExporter.Domain.Exporting;
@@ -12,39 +12,31 @@ namespace DiscordChatExporter.Cli.Commands.Base
{
public abstract class ExportCommandBase : TokenCommandBase
{
[CommandOption("output", 'o',
Description = "Output file or directory path.")]
public string OutputPath { get; set; } = Directory.GetCurrentDirectory();
[CommandOption("output", 'o', Description = "Output file or directory path.")]
public string OutputPath { get; init; } = Directory.GetCurrentDirectory();
[CommandOption("format", 'f',
Description = "Export format.")]
public ExportFormat ExportFormat { get; set; } = ExportFormat.HtmlDark;
[CommandOption("format", 'f', Description = "Export format.")]
public ExportFormat ExportFormat { get; init; } = ExportFormat.HtmlDark;
[CommandOption("after",
Description = "Only include messages sent after this date.")]
public DateTimeOffset? After { get; set; }
[CommandOption("after", Description = "Only include messages sent after this date or message ID.")]
public Snowflake? After { get; init; }
[CommandOption("before",
Description = "Only include messages sent before this date.")]
public DateTimeOffset? Before { get; set; }
[CommandOption("before", Description = "Only include messages sent before this date or message ID.")]
public Snowflake? Before { get; init; }
[CommandOption("partition", 'p',
Description = "Split output into partitions limited to this number of messages.")]
public int? PartitionLimit { get; set; }
[CommandOption("partition", 'p', Description = "Split output into partitions limited to this number of messages.")]
public int? PartitionLimit { get; init; }
[CommandOption("media",
Description = "Download referenced media content.")]
public bool ShouldDownloadMedia { get; set; }
[CommandOption("media", Description = "Download referenced media content.")]
public bool ShouldDownloadMedia { get; init; }
[CommandOption("reuse-media",
Description = "Reuse already existing media content to skip redundant downloads.")]
public bool ShouldReuseMedia { get; set; }
[CommandOption("reuse-media", Description = "Reuse already existing media content to skip redundant downloads.")]
public bool ShouldReuseMedia { get; init; }
[CommandOption("dateformat",
Description = "Format used when writing dates.")]
public string DateFormat { get; set; } = "dd-MMM-yy hh:mm tt";
[CommandOption("dateformat", Description = "Format used when writing dates.")]
public string DateFormat { get; init; } = "dd-MMM-yy hh:mm tt";
protected ChannelExporter GetChannelExporter() => new ChannelExporter(GetDiscordClient());
protected ChannelExporter GetChannelExporter() => new(GetDiscordClient());
protected async ValueTask ExportAsync(IConsole console, Guild guild, Channel channel)
{
@@ -76,7 +68,7 @@ namespace DiscordChatExporter.Cli.Commands.Base
await ExportAsync(console, guild, channel);
}
protected async ValueTask ExportAsync(IConsole console, string channelId)
protected async ValueTask ExportAsync(IConsole console, Snowflake channelId)
{
var channel = await GetDiscordClient().GetChannelAsync(channelId);
await ExportAsync(console, channel);
@@ -86,9 +78,7 @@ namespace DiscordChatExporter.Cli.Commands.Base
{
if (ShouldReuseMedia && !ShouldDownloadMedia)
{
throw new CommandException(
"The --reuse-media option cannot be used without the --media option."
);
throw new CommandException("The --reuse-media option cannot be used without the --media option.");
}
return default;
@@ -17,9 +17,8 @@ namespace DiscordChatExporter.Cli.Commands.Base
{
public abstract class ExportMultipleCommandBase : ExportCommandBase
{
[CommandOption("parallel",
Description = "Limits how many channels can be exported in parallel.")]
public int ParallelLimit { get; set; } = 1;
[CommandOption("parallel", Description = "Limits how many channels can be exported in parallel.")]
public int ParallelLimit { get; init; } = 1;
protected async ValueTask ExportMultipleAsync(IConsole console, IReadOnlyList<Channel> channels)
{
@@ -32,7 +31,7 @@ namespace DiscordChatExporter.Cli.Commands.Base
var operations = progress.Wrap().CreateOperations(channels.Count);
var successfulExportCount = 0;
var errors = new ConcurrentBag<string>();
var errors = new ConcurrentBag<(Channel, string)>();
await channels.Zip(operations).ParallelForEachAsync(async tuple =>
{
@@ -61,7 +60,7 @@ namespace DiscordChatExporter.Cli.Commands.Base
}
catch (DiscordChatExporterException ex) when (!ex.IsCritical)
{
errors.Add(ex.Message);
errors.Add((channel, ex.Message));
}
finally
{
@@ -71,8 +70,8 @@ namespace DiscordChatExporter.Cli.Commands.Base
console.Output.WriteLine();
foreach (var error in errors)
console.Error.WriteLine(error);
foreach (var (channel, error) in errors)
console.Error.WriteLine($"Channel '{channel}': {error}");
console.Output.WriteLine($"Successfully exported {successfulExportCount} channel(s).");
}
@@ -7,24 +7,20 @@ namespace DiscordChatExporter.Cli.Commands.Base
{
public abstract class TokenCommandBase : ICommand
{
[CommandOption("token", 't', IsRequired = true,
EnvironmentVariableName = "DISCORD_TOKEN",
Description = "Authorization token.")]
public string TokenValue { get; set; } = "";
[CommandOption("token", 't', IsRequired = true, EnvironmentVariableName = "DISCORD_TOKEN", Description = "Authorization token.")]
public string TokenValue { get; init; } = "";
[CommandOption("bot", 'b',
EnvironmentVariableName = "DISCORD_TOKEN_BOT",
Description = "Authorize as a bot.")]
public bool IsBotToken { get; set; }
[CommandOption("bot", 'b', EnvironmentVariableName = "DISCORD_TOKEN_BOT", Description = "Authorize as a bot.")]
public bool IsBotToken { get; init; }
protected AuthToken GetAuthToken() => new AuthToken(
protected AuthToken GetAuthToken() => new(
IsBotToken
? AuthTokenType.Bot
: AuthTokenType.User,
TokenValue
);
protected DiscordClient GetDiscordClient() => new DiscordClient(GetAuthToken());
protected DiscordClient GetDiscordClient() => new(GetAuthToken());
public abstract ValueTask ExecuteAsync(IConsole console);
}
@@ -10,9 +10,8 @@ namespace DiscordChatExporter.Cli.Commands
[Command("exportall", Description = "Export all accessible channels.")]
public class ExportAllCommand : ExportMultipleCommandBase
{
[CommandOption("include-dm",
Description = "Include direct message channels.")]
public bool IncludeDirectMessages { get; set; } = true;
[CommandOption("include-dm", Description = "Include direct message channels.")]
public bool IncludeDirectMessages { get; init; } = true;
public override async ValueTask ExecuteAsync(IConsole console)
{
@@ -2,15 +2,15 @@
using CliFx;
using CliFx.Attributes;
using DiscordChatExporter.Cli.Commands.Base;
using DiscordChatExporter.Domain.Discord;
namespace DiscordChatExporter.Cli.Commands
{
[Command("export", Description = "Export a channel.")]
public class ExportChannelCommand : ExportCommandBase
{
[CommandOption("channel", 'c', IsRequired = true,
Description = "Channel ID.")]
public string ChannelId { get; set; } = "";
[CommandOption("channel", 'c', IsRequired = true, Description = "Channel ID.")]
public Snowflake ChannelId { get; init; }
public override async ValueTask ExecuteAsync(IConsole console)
{
@@ -2,6 +2,7 @@
using CliFx;
using CliFx.Attributes;
using DiscordChatExporter.Cli.Commands.Base;
using DiscordChatExporter.Domain.Discord;
using DiscordChatExporter.Domain.Utilities;
namespace DiscordChatExporter.Cli.Commands
@@ -9,9 +10,8 @@ namespace DiscordChatExporter.Cli.Commands
[Command("exportguild", Description = "Export all channels within specified guild.")]
public class ExportGuildCommand : ExportMultipleCommandBase
{
[CommandOption("guild", 'g', IsRequired = true,
Description = "Guild ID.")]
public string GuildId { get; set; } = "";
[CommandOption("guild", 'g', IsRequired = true, Description = "Guild ID.")]
public Snowflake GuildId { get; init; }
public override async ValueTask ExecuteAsync(IConsole console)
{
@@ -3,6 +3,7 @@ using System.Threading.Tasks;
using CliFx;
using CliFx.Attributes;
using DiscordChatExporter.Cli.Commands.Base;
using DiscordChatExporter.Domain.Discord;
using DiscordChatExporter.Domain.Utilities;
namespace DiscordChatExporter.Cli.Commands
@@ -10,9 +11,8 @@ namespace DiscordChatExporter.Cli.Commands
[Command("channels", Description = "Get the list of channels in a guild.")]
public class GetChannelsCommand : TokenCommandBase
{
[CommandOption("guild", 'g', IsRequired = true,
Description = "Guild ID.")]
public string GuildId { get; set; } = "";
[CommandOption("guild", 'g', IsRequired = true, Description = "Guild ID.")]
public Snowflake GuildId { get; init; }
public override async ValueTask ExecuteAsync(IConsole console)
{
@@ -1,5 +1,4 @@
<Project Sdk="Microsoft.NET.Sdk">
<Import Project="../DiscordChatExporter.props" />
<PropertyGroup>
<OutputType>Exe</OutputType>
@@ -7,7 +6,7 @@
</PropertyGroup>
<ItemGroup>
<PackageReference Include="CliFx" Version="1.5.0" />
<PackageReference Include="CliFx" Version="1.6.0" />
<PackageReference Include="Gress" Version="1.2.0" />
<PackageReference Include="Tyrrrz.Extensions" Version="1.6.5" />
</ItemGroup>
@@ -0,0 +1,9 @@
// ReSharper disable CheckNamespace
// TODO: remove after moving to .NET 5
namespace System.Runtime.CompilerServices
{
internal static class IsExternalInit
{
}
}
@@ -8,7 +8,9 @@ using System.Threading.Tasks;
using DiscordChatExporter.Domain.Discord.Models;
using DiscordChatExporter.Domain.Exceptions;
using DiscordChatExporter.Domain.Internal;
using DiscordChatExporter.Domain.Internal.Extensions;
using DiscordChatExporter.Domain.Utilities;
using JsonExtensions.Http;
using JsonExtensions.Reading;
namespace DiscordChatExporter.Domain.Discord
{
@@ -17,7 +19,7 @@ namespace DiscordChatExporter.Domain.Discord
private readonly HttpClient _httpClient;
private readonly AuthToken _token;
private readonly Uri _baseUri = new Uri("https://discord.com/api/v6/", UriKind.Absolute);
private readonly Uri _baseUri = new("https://discord.com/api/v8/", UriKind.Absolute);
public DiscordClient(HttpClient httpClient, AuthToken token)
{
@@ -68,13 +70,14 @@ namespace DiscordChatExporter.Domain.Discord
{
yield return Guild.DirectMessages;
var afterId = "";
var currentAfter = Snowflake.Zero;
while (true)
{
var url = new UrlBuilder()
.SetPath("users/@me/guilds")
.SetQueryParameter("limit", "100")
.SetQueryParameter("after", afterId)
.SetQueryParameter("after", currentAfter.ToString())
.Build();
var response = await GetJsonResponseAsync(url);
@@ -84,7 +87,7 @@ namespace DiscordChatExporter.Domain.Discord
{
yield return guild;
afterId = guild.Id;
currentAfter = guild.Id;
isEmpty = false;
}
@@ -93,7 +96,7 @@ namespace DiscordChatExporter.Domain.Discord
}
}
public async ValueTask<Guild> GetGuildAsync(string guildId)
public async ValueTask<Guild> GetGuildAsync(Snowflake guildId)
{
if (guildId == Guild.DirectMessages.Id)
return Guild.DirectMessages;
@@ -102,7 +105,7 @@ namespace DiscordChatExporter.Domain.Discord
return Guild.Parse(response);
}
public async IAsyncEnumerable<Channel> GetGuildChannelsAsync(string guildId)
public async IAsyncEnumerable<Channel> GetGuildChannelsAsync(Snowflake guildId)
{
if (guildId == Guild.DirectMessages.Id)
{
@@ -139,7 +142,7 @@ namespace DiscordChatExporter.Domain.Discord
}
}
public async IAsyncEnumerable<Role> GetGuildRolesAsync(string guildId)
public async IAsyncEnumerable<Role> GetGuildRolesAsync(Snowflake guildId)
{
if (guildId == Guild.DirectMessages.Id)
yield break;
@@ -150,7 +153,7 @@ namespace DiscordChatExporter.Domain.Discord
yield return Role.Parse(roleJson);
}
public async ValueTask<Member?> TryGetGuildMemberAsync(string guildId, User user)
public async ValueTask<Member?> TryGetGuildMemberAsync(Snowflake guildId, User user)
{
if (guildId == Guild.DirectMessages.Id)
return Member.CreateForUser(user);
@@ -159,30 +162,31 @@ namespace DiscordChatExporter.Domain.Discord
return response?.Pipe(Member.Parse);
}
private async ValueTask<string> GetChannelCategoryAsync(string channelParentId)
private async ValueTask<string> GetChannelCategoryAsync(Snowflake channelParentId)
{
var response = await GetJsonResponseAsync($"channels/{channelParentId}");
return response.GetProperty("name").GetString();
}
public async ValueTask<Channel> GetChannelAsync(string channelId)
public async ValueTask<Channel> GetChannelAsync(Snowflake channelId)
{
var response = await GetJsonResponseAsync($"channels/{channelId}");
var parentId = response.GetPropertyOrNull("parent_id")?.GetString();
var category = !string.IsNullOrWhiteSpace(parentId)
? await GetChannelCategoryAsync(parentId)
var parentId = response.GetPropertyOrNull("parent_id")?.GetString().Pipe(Snowflake.Parse);
var category = parentId != null
? await GetChannelCategoryAsync(parentId.Value)
: null;
return Channel.Parse(response, category);
}
private async ValueTask<Message?> TryGetLastMessageAsync(string channelId, DateTimeOffset? before = null)
private async ValueTask<Message?> TryGetLastMessageAsync(Snowflake channelId, Snowflake? before = null)
{
var url = new UrlBuilder()
.SetPath($"channels/{channelId}/messages")
.SetQueryParameter("limit", "1")
.SetQueryParameter("before", before?.ToSnowflake())
.SetQueryParameter("before", before?.ToString())
.Build();
var response = await GetJsonResponseAsync(url);
@@ -190,9 +194,9 @@ namespace DiscordChatExporter.Domain.Discord
}
public async IAsyncEnumerable<Message> GetMessagesAsync(
string channelId,
DateTimeOffset? after = null,
DateTimeOffset? before = null,
Snowflake channelId,
Snowflake? after = null,
Snowflake? before = null,
IProgress<double>? progress = null)
{
// Get the last message in the specified range.
@@ -200,19 +204,19 @@ namespace DiscordChatExporter.Domain.Discord
// will not appear in the output.
// Additionally, it provides the date of the last message, which is used to calculate progress.
var lastMessage = await TryGetLastMessageAsync(channelId, before);
if (lastMessage == null || lastMessage.Timestamp < after)
if (lastMessage == null || lastMessage.Timestamp < after?.ToDate())
yield break;
// Keep track of first message in range in order to calculate progress
var firstMessage = default(Message);
var afterId = after?.ToSnowflake() ?? "0";
var currentAfter = after ?? Snowflake.Zero;
while (true)
{
var url = new UrlBuilder()
.SetPath($"channels/{channelId}/messages")
.SetQueryParameter("limit", "100")
.SetQueryParameter("after", afterId)
.SetQueryParameter("after", currentAfter.ToString())
.Build();
var response = await GetJsonResponseAsync(url);
@@ -242,9 +246,9 @@ namespace DiscordChatExporter.Domain.Discord
);
yield return message;
afterId = message.Id;
currentAfter = message.Id;
}
}
}
}
}
}
@@ -3,14 +3,15 @@ using System.Collections.Generic;
using System.IO;
using System.Text.Json;
using DiscordChatExporter.Domain.Discord.Models.Common;
using DiscordChatExporter.Domain.Internal.Extensions;
using DiscordChatExporter.Domain.Utilities;
using JsonExtensions.Reading;
namespace DiscordChatExporter.Domain.Discord.Models
{
// https://discord.com/developers/docs/resources/channel#attachment-object
public partial class Attachment : IHasId
{
public string Id { get; }
public Snowflake Id { get; }
public string Url { get; }
@@ -31,7 +32,7 @@ namespace DiscordChatExporter.Domain.Discord.Models
public FileSize FileSize { get; }
public Attachment(string id, string url, string fileName, int? width, int? height, FileSize fileSize)
public Attachment(Snowflake id, string url, string fileName, int? width, int? height, FileSize fileSize)
{
Id = id;
Url = url;
@@ -46,21 +47,18 @@ namespace DiscordChatExporter.Domain.Discord.Models
public partial class Attachment
{
private static readonly HashSet<string> ImageFileExtensions =
new HashSet<string>(StringComparer.OrdinalIgnoreCase)
{".jpg", ".jpeg", ".png", ".gif", ".bmp", ".webp"};
private static readonly HashSet<string> ImageFileExtensions = new(StringComparer.OrdinalIgnoreCase)
{".jpg", ".jpeg", ".png", ".gif", ".bmp", ".webp"};
private static readonly HashSet<string> VideoFileExtensions =
new HashSet<string>(StringComparer.OrdinalIgnoreCase)
{".mp4", ".webm"};
private static readonly HashSet<string> VideoFileExtensions = new(StringComparer.OrdinalIgnoreCase)
{".mp4", ".webm"};
private static readonly HashSet<string> AudioFileExtensions =
new HashSet<string>(StringComparer.OrdinalIgnoreCase)
{".mp3", ".wav", ".ogg", ".flac", ".m4a"};
private static readonly HashSet<string> AudioFileExtensions = new(StringComparer.OrdinalIgnoreCase)
{".mp3", ".wav", ".ogg", ".flac", ".m4a"};
public static Attachment Parse(JsonElement json)
{
var id = json.GetProperty("id").GetString();
var id = json.GetProperty("id").GetString().Pipe(Snowflake.Parse);
var url = json.GetProperty("url").GetString();
var width = json.GetPropertyOrNull("width")?.GetInt32();
var height = json.GetPropertyOrNull("height")?.GetInt32();
@@ -1,7 +1,8 @@
using System.Linq;
using System.Text.Json;
using DiscordChatExporter.Domain.Discord.Models.Common;
using DiscordChatExporter.Domain.Internal.Extensions;
using DiscordChatExporter.Domain.Utilities;
using JsonExtensions.Reading;
using Tyrrrz.Extensions;
namespace DiscordChatExporter.Domain.Discord.Models
@@ -22,7 +23,7 @@ namespace DiscordChatExporter.Domain.Discord.Models
// https://discord.com/developers/docs/resources/channel#channel-object
public partial class Channel : IHasId
{
public string Id { get; }
public Snowflake Id { get; }
public ChannelType Type { get; }
@@ -33,7 +34,7 @@ namespace DiscordChatExporter.Domain.Discord.Models
Type == ChannelType.GuildNews ||
Type == ChannelType.GuildStore;
public string GuildId { get; }
public Snowflake GuildId { get; }
public string Category { get; }
@@ -41,7 +42,7 @@ namespace DiscordChatExporter.Domain.Discord.Models
public string? Topic { get; }
public Channel(string id, ChannelType type, string guildId, string category, string name, string? topic)
public Channel(Snowflake id, ChannelType type, Snowflake guildId, string category, string name, string? topic)
{
Id = id;
Type = type;
@@ -68,8 +69,8 @@ namespace DiscordChatExporter.Domain.Discord.Models
public static Channel Parse(JsonElement json, string? category = null)
{
var id = json.GetProperty("id").GetString();
var guildId = json.GetPropertyOrNull("guild_id")?.GetString();
var id = json.GetProperty("id").GetString().Pipe(Snowflake.Parse);
var guildId = json.GetPropertyOrNull("guild_id")?.GetString().Pipe(Snowflake.Parse);
var topic = json.GetPropertyOrNull("topic")?.GetString();
var type = (ChannelType) json.GetProperty("type").GetInt32();
@@ -77,7 +78,7 @@ namespace DiscordChatExporter.Domain.Discord.Models
var name =
json.GetPropertyOrNull("name")?.GetString() ??
json.GetPropertyOrNull("recipients")?.EnumerateArray().Select(User.Parse).Select(u => u.Name).JoinToString(", ") ??
id;
id.ToString();
return new Channel(
id,
@@ -52,7 +52,7 @@ namespace DiscordChatExporter.Domain.Discord.Models.Common
if (Math.Abs(TotalKiloBytes) >= 1)
return "KB";
return "B";
return "bytes";
}
public override string ToString() => $"{GetLargestWholeNumberValue():0.##} {GetLargestWholeNumberSymbol()}";
@@ -60,6 +60,6 @@ namespace DiscordChatExporter.Domain.Discord.Models.Common
public partial struct FileSize
{
public static FileSize FromBytes(long bytes) => new FileSize(bytes);
public static FileSize FromBytes(long bytes) => new(bytes);
}
}
@@ -2,6 +2,6 @@
{
public interface IHasId
{
string Id { get; }
Snowflake Id { get; }
}
}
@@ -5,13 +5,13 @@ namespace DiscordChatExporter.Domain.Discord.Models.Common
{
public partial class IdBasedEqualityComparer : IEqualityComparer<IHasId>
{
public bool Equals(IHasId? x, IHasId? y) => StringComparer.Ordinal.Equals(x?.Id, y?.Id);
public bool Equals(IHasId? x, IHasId? y) => x?.Id == y?.Id;
public int GetHashCode(IHasId obj) => StringComparer.Ordinal.GetHashCode(obj.Id);
public int GetHashCode(IHasId obj) => obj.Id.GetHashCode();
}
public partial class IdBasedEqualityComparer
{
public static IdBasedEqualityComparer Instance { get; } = new IdBasedEqualityComparer();
public static IdBasedEqualityComparer Instance { get; } = new();
}
}
@@ -4,6 +4,8 @@ using System.Drawing;
using System.Linq;
using System.Text.Json;
using DiscordChatExporter.Domain.Internal.Extensions;
using DiscordChatExporter.Domain.Utilities;
using JsonExtensions.Reading;
namespace DiscordChatExporter.Domain.Discord.Models
{
@@ -1,5 +1,5 @@
using System.Text.Json;
using DiscordChatExporter.Domain.Internal.Extensions;
using JsonExtensions.Reading;
namespace DiscordChatExporter.Domain.Discord.Models
{
@@ -1,5 +1,5 @@
using System.Text.Json;
using DiscordChatExporter.Domain.Internal.Extensions;
using JsonExtensions.Reading;
namespace DiscordChatExporter.Domain.Discord.Models
{
@@ -1,5 +1,5 @@
using System.Text.Json;
using DiscordChatExporter.Domain.Internal.Extensions;
using JsonExtensions.Reading;
namespace DiscordChatExporter.Domain.Discord.Models
{
@@ -1,5 +1,5 @@
using System.Text.Json;
using DiscordChatExporter.Domain.Internal.Extensions;
using JsonExtensions.Reading;
namespace DiscordChatExporter.Domain.Discord.Models
{
@@ -2,7 +2,7 @@
using System.Linq;
using System.Text;
using System.Text.Json;
using DiscordChatExporter.Domain.Internal.Extensions;
using JsonExtensions.Reading;
using Tyrrrz.Extensions;
namespace DiscordChatExporter.Domain.Discord.Models
@@ -1,18 +1,19 @@
using System.Text.Json;
using DiscordChatExporter.Domain.Discord.Models.Common;
using DiscordChatExporter.Domain.Utilities;
namespace DiscordChatExporter.Domain.Discord.Models
{
// https://discord.com/developers/docs/resources/guild#guild-object
public partial class Guild : IHasId
{
public string Id { get; }
public Snowflake Id { get; }
public string Name { get; }
public string IconUrl { get; }
public Guild(string id, string name, string iconUrl)
public Guild(Snowflake id, string name, string iconUrl)
{
Id = id;
Name = name;
@@ -24,18 +25,17 @@ namespace DiscordChatExporter.Domain.Discord.Models
public partial class Guild
{
public static Guild DirectMessages { get; } =
new Guild("@me", "Direct Messages", GetDefaultIconUrl());
public static Guild DirectMessages { get; } = new(Snowflake.Zero, "Direct Messages", GetDefaultIconUrl());
private static string GetDefaultIconUrl() =>
"https://cdn.discordapp.com/embed/avatars/0.png";
private static string GetIconUrl(string id, string iconHash) =>
private static string GetIconUrl(Snowflake id, string iconHash) =>
$"https://cdn.discordapp.com/icons/{id}/{iconHash}.png";
public static Guild Parse(JsonElement json)
{
var id = json.GetProperty("id").GetString();
var id = json.GetProperty("id").GetString().Pipe(Snowflake.Parse);
var name = json.GetProperty("name").GetString();
var iconHash = json.GetProperty("icon").GetString();
@@ -3,22 +3,23 @@ using System.Collections.Generic;
using System.Linq;
using System.Text.Json;
using DiscordChatExporter.Domain.Discord.Models.Common;
using DiscordChatExporter.Domain.Internal.Extensions;
using DiscordChatExporter.Domain.Utilities;
using JsonExtensions.Reading;
namespace DiscordChatExporter.Domain.Discord.Models
{
// https://discord.com/developers/docs/resources/guild#guild-member-object
public partial class Member : IHasId
{
public string Id => User.Id;
public Snowflake Id => User.Id;
public User User { get; }
public string Nick { get; }
public IReadOnlyList<string> RoleIds { get; }
public IReadOnlyList<Snowflake> RoleIds { get; }
public Member(User user, string? nick, IReadOnlyList<string> roleIds)
public Member(User user, string? nick, IReadOnlyList<Snowflake> roleIds)
{
User = user;
Nick = nick ?? user.Name;
@@ -30,8 +31,7 @@ namespace DiscordChatExporter.Domain.Discord.Models
public partial class Member
{
public static Member CreateForUser(User user) =>
new Member(user, null, Array.Empty<string>());
public static Member CreateForUser(User user) => new(user, null, Array.Empty<Snowflake>());
public static Member Parse(JsonElement json)
{
@@ -39,8 +39,8 @@ namespace DiscordChatExporter.Domain.Discord.Models
var nick = json.GetPropertyOrNull("nick")?.GetString();
var roleIds =
json.GetPropertyOrNull("roles")?.EnumerateArray().Select(j => j.GetString()).ToArray() ??
Array.Empty<string>();
json.GetPropertyOrNull("roles")?.EnumerateArray().Select(j => j.GetString().Pipe(Snowflake.Parse)).ToArray() ??
Array.Empty<Snowflake>();
return new Member(
user,
@@ -3,27 +3,29 @@ using System.Collections.Generic;
using System.Linq;
using System.Text.Json;
using DiscordChatExporter.Domain.Discord.Models.Common;
using DiscordChatExporter.Domain.Internal.Extensions;
using DiscordChatExporter.Domain.Utilities;
using JsonExtensions.Reading;
namespace DiscordChatExporter.Domain.Discord.Models
{
// https://discord.com/developers/docs/resources/channel#message-object-message-types
public enum MessageType
{
Default,
RecipientAdd,
RecipientRemove,
Call,
ChannelNameChange,
ChannelIconChange,
ChannelPinnedMessage,
GuildMemberJoin
Default = 0,
RecipientAdd = 1,
RecipientRemove = 2,
Call = 3,
ChannelNameChange = 4,
ChannelIconChange = 5,
ChannelPinnedMessage = 6,
GuildMemberJoin = 7,
Reply = 19
}
// https://discord.com/developers/docs/resources/channel#message-object
public partial class Message : IHasId
{
public string Id { get; }
public Snowflake Id { get; }
public MessageType Type { get; }
@@ -47,8 +49,12 @@ namespace DiscordChatExporter.Domain.Discord.Models
public IReadOnlyList<User> MentionedUsers { get; }
public MessageReference? Reference { get; }
public Message? ReferencedMessage { get; }
public Message(
string id,
Snowflake id,
MessageType type,
User author,
DateTimeOffset timestamp,
@@ -59,7 +65,9 @@ namespace DiscordChatExporter.Domain.Discord.Models
IReadOnlyList<Attachment> attachments,
IReadOnlyList<Embed> embeds,
IReadOnlyList<Reaction> reactions,
IReadOnlyList<User> mentionedUsers)
IReadOnlyList<User> mentionedUsers,
MessageReference? messageReference,
Message? referencedMessage)
{
Id = id;
Type = type;
@@ -73,6 +81,8 @@ namespace DiscordChatExporter.Domain.Discord.Models
Embeds = embeds;
Reactions = reactions;
MentionedUsers = mentionedUsers;
Reference = messageReference;
ReferencedMessage = referencedMessage;
}
public override string ToString() => Content;
@@ -82,20 +92,22 @@ namespace DiscordChatExporter.Domain.Discord.Models
{
public static Message Parse(JsonElement json)
{
var id = json.GetProperty("id").GetString();
var id = json.GetProperty("id").GetString().Pipe(Snowflake.Parse);
var author = json.GetProperty("author").Pipe(User.Parse);
var timestamp = json.GetProperty("timestamp").GetDateTimeOffset();
var editedTimestamp = json.GetPropertyOrNull("edited_timestamp")?.GetDateTimeOffset();
var callEndedTimestamp = json.GetPropertyOrNull("call")?.GetPropertyOrNull("ended_timestamp")?.GetDateTimeOffset();
var type = (MessageType) json.GetProperty("type").GetInt32();
var isPinned = json.GetPropertyOrNull("pinned")?.GetBoolean() ?? false;
var messageReference = json.GetPropertyOrNull("message_reference")?.Pipe(MessageReference.Parse);
var referencedMessage = json.GetPropertyOrNull("referenced_message")?.Pipe(Message.Parse);
var content = type switch
{
MessageType.RecipientAdd => "Added a recipient.",
MessageType.RecipientRemove => "Removed a recipient.",
MessageType.Call =>
$"Started a call that lasted {callEndedTimestamp?.Pipe(t => t - timestamp).Pipe(t => (int) t.TotalMinutes) ?? 0} minutes.",
$"Started a call that lasted {callEndedTimestamp?.Pipe(t => t - timestamp).Pipe(t => (int) t.TotalMinutes) ?? 0} minutes.",
MessageType.ChannelNameChange => "Changed the channel name.",
MessageType.ChannelIconChange => "Changed the channel icon.",
MessageType.ChannelPinnedMessage => "Pinned a message.",
@@ -131,7 +143,9 @@ namespace DiscordChatExporter.Domain.Discord.Models
attachments,
embeds,
reactions,
mentionedUsers
mentionedUsers,
messageReference,
referencedMessage
);
}
}
@@ -0,0 +1,37 @@
using System.Text.Json;
using DiscordChatExporter.Domain.Utilities;
using JsonExtensions.Reading;
namespace DiscordChatExporter.Domain.Discord.Models
{
// https://discord.com/developers/docs/resources/channel#message-object-message-reference-structure
public partial class MessageReference
{
public Snowflake? MessageId { get; }
public Snowflake? ChannelId { get; }
public Snowflake? GuildId { get; }
public MessageReference(Snowflake? messageId, Snowflake? channelId, Snowflake? guildId)
{
MessageId = messageId;
ChannelId = channelId;
GuildId = guildId;
}
public override string ToString() => MessageId?.ToString() ?? "<unknown reference>";
}
public partial class MessageReference
{
public static MessageReference Parse(JsonElement json)
{
var messageId = json.GetPropertyOrNull("message_id")?.GetString().Pipe(Snowflake.Parse);
var channelId = json.GetPropertyOrNull("channel_id")?.GetString().Pipe(Snowflake.Parse);
var guildId = json.GetPropertyOrNull("guild_id")?.GetString().Pipe(Snowflake.Parse);
return new MessageReference(messageId, channelId, guildId);
}
}
}
@@ -1,5 +1,5 @@
using System.Text.Json;
using DiscordChatExporter.Domain.Internal.Extensions;
using DiscordChatExporter.Domain.Utilities;
namespace DiscordChatExporter.Domain.Discord.Models
{
@@ -1,13 +1,16 @@
using System.Drawing;
using System.Text.Json;
using DiscordChatExporter.Domain.Discord.Models.Common;
using DiscordChatExporter.Domain.Internal.Extensions;
using DiscordChatExporter.Domain.Utilities;
using JsonExtensions.Reading;
namespace DiscordChatExporter.Domain.Discord.Models
{
// https://discord.com/developers/docs/topics/permissions#role-object
public partial class Role
public partial class Role : IHasId
{
public string Id { get; }
public Snowflake Id { get; }
public string Name { get; }
@@ -15,7 +18,7 @@ namespace DiscordChatExporter.Domain.Discord.Models
public Color? Color { get; }
public Role(string id, string name, int position, Color? color)
public Role(Snowflake id, string name, int position, Color? color)
{
Id = id;
Name = name;
@@ -30,7 +33,7 @@ namespace DiscordChatExporter.Domain.Discord.Models
{
public static Role Parse(JsonElement json)
{
var id = json.GetProperty("id").GetString();
var id = json.GetProperty("id").GetString().Pipe(Snowflake.Parse);
var name = json.GetProperty("name").GetString();
var position = json.GetProperty("position").GetInt32();
@@ -1,14 +1,15 @@
using System;
using System.Text.Json;
using DiscordChatExporter.Domain.Discord.Models.Common;
using DiscordChatExporter.Domain.Internal.Extensions;
using DiscordChatExporter.Domain.Utilities;
using JsonExtensions.Reading;
namespace DiscordChatExporter.Domain.Discord.Models
{
// https://discord.com/developers/docs/resources/user#user-object
public partial class User : IHasId
{
public string Id { get; }
public Snowflake Id { get; }
public bool IsBot { get; }
@@ -20,7 +21,7 @@ namespace DiscordChatExporter.Domain.Discord.Models
public string AvatarUrl { get; }
public User(string id, bool isBot, int discriminator, string name, string avatarUrl)
public User(Snowflake id, bool isBot, int discriminator, string name, string avatarUrl)
{
Id = id;
IsBot = isBot;
@@ -37,7 +38,7 @@ namespace DiscordChatExporter.Domain.Discord.Models
private static string GetDefaultAvatarUrl(int discriminator) =>
$"https://cdn.discordapp.com/embed/avatars/{discriminator % 5}.png";
private static string GetAvatarUrl(string id, string avatarHash)
private static string GetAvatarUrl(Snowflake id, string avatarHash)
{
// Animated
if (avatarHash.StartsWith("a_", StringComparison.Ordinal))
@@ -49,7 +50,7 @@ namespace DiscordChatExporter.Domain.Discord.Models
public static User Parse(JsonElement json)
{
var id = json.GetProperty("id").GetString();
var id = json.GetProperty("id").GetString().Pipe(Snowflake.Parse);
var discriminator = json.GetProperty("discriminator").GetString().Pipe(int.Parse);
var name = json.GetProperty("username").GetString();
var avatarHash = json.GetProperty("avatar").GetString();
@@ -0,0 +1,68 @@
using System;
using System.Globalization;
using System.Text.RegularExpressions;
namespace DiscordChatExporter.Domain.Discord
{
public readonly partial struct Snowflake
{
public ulong Value { get; }
public Snowflake(ulong value) => Value = value;
public DateTimeOffset ToDate() =>
DateTimeOffset.FromUnixTimeMilliseconds((long) ((Value >> 22) + 1420070400000UL)).ToLocalTime();
public override string ToString() => Value.ToString(CultureInfo.InvariantCulture);
}
public partial struct Snowflake
{
public static Snowflake Zero { get; } = new(0);
public static Snowflake FromDate(DateTimeOffset date)
{
var value = ((ulong) date.ToUnixTimeMilliseconds() - 1420070400000UL) << 22;
return new Snowflake(value);
}
public static Snowflake? TryParse(string? str, IFormatProvider? formatProvider = null)
{
if (string.IsNullOrWhiteSpace(str))
return null;
// As number
if (Regex.IsMatch(str, @"^\d{15,}$") &&
ulong.TryParse(str, NumberStyles.Number, formatProvider, out var value))
{
return new Snowflake(value);
}
// As date
if (DateTimeOffset.TryParse(str, formatProvider, DateTimeStyles.None, out var date))
{
return FromDate(date);
}
return null;
}
public static Snowflake Parse(string str, IFormatProvider? formatProvider) =>
TryParse(str, formatProvider) ?? throw new FormatException($"Invalid snowflake: {str}.");
public static Snowflake Parse(string str) => Parse(str, null);
}
public partial struct Snowflake : IEquatable<Snowflake>
{
public bool Equals(Snowflake other) => Value == other.Value;
public override bool Equals(object? obj) => obj is Snowflake other && Equals(other);
public override int GetHashCode() => Value.GetHashCode();
public static bool operator ==(Snowflake left, Snowflake right) => left.Equals(right);
public static bool operator !=(Snowflake left, Snowflake right) => !(left == right);
}
}
@@ -1,8 +1,12 @@
<Project Sdk="Microsoft.NET.Sdk">
<Import Project="../DiscordChatExporter.props" />
<PropertyGroup>
<EmitCompilerGeneratedFiles>true</EmitCompilerGeneratedFiles>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="MiniRazor" Version="1.1.0" />
<PackageReference Include="JsonExtensions" Version="1.0.1" />
<PackageReference Include="MiniRazor" Version="2.0.3" />
<PackageReference Include="Polly" Version="7.2.1" />
<PackageReference Include="Tyrrrz.Extensions" Version="1.6.5" />
</ItemGroup>
@@ -11,9 +15,9 @@
<EmbeddedResource Include="Exporting\Writers\Html\Core.css" />
<EmbeddedResource Include="Exporting\Writers\Html\Dark.css" />
<EmbeddedResource Include="Exporting\Writers\Html\Light.css" />
<EmbeddedResource Include="Exporting\Writers\Html\PreambleTemplate.cshtml" />
<EmbeddedResource Include="Exporting\Writers\Html\PostambleTemplate.cshtml" />
<EmbeddedResource Include="Exporting\Writers\Html\MessageGroupTemplate.cshtml" />
<AdditionalFiles Include="Exporting\Writers\Html\PreambleTemplate.cshtml" IsRazorTemplate="true" />
<AdditionalFiles Include="Exporting\Writers\Html\PostambleTemplate.cshtml" IsRazorTemplate="true" />
<AdditionalFiles Include="Exporting\Writers\Html\MessageGroupTemplate.cshtml" IsRazorTemplate="true" />
</ItemGroup>
</Project>
@@ -46,9 +46,9 @@ Failed to perform an HTTP request.
return new DiscordChatExporterException(message);
}
internal static DiscordChatExporterException ChannelIsEmpty(string channel)
internal static DiscordChatExporterException ChannelIsEmpty()
{
var message = $"Channel '{channel}' contains no messages for the specified period.";
var message = $"No messages for the specified period.";
return new DiscordChatExporterException(message);
}
}
@@ -59,7 +59,7 @@ namespace DiscordChatExporter.Domain.Exporting
// Throw if no messages were exported
if (!exportedAnything)
throw DiscordChatExporterException.ChannelIsEmpty(request.Channel.Name);
throw DiscordChatExporterException.ChannelIsEmpty();
}
}
}
@@ -5,6 +5,7 @@ using System.IO;
using System.Linq;
using System.Net.Http;
using System.Threading.Tasks;
using DiscordChatExporter.Domain.Discord;
using DiscordChatExporter.Domain.Discord.Models;
using DiscordChatExporter.Domain.Internal.Extensions;
using Tyrrrz.Extensions;
@@ -44,16 +45,13 @@ namespace DiscordChatExporter.Domain.Exporting
var dateFormat => date.ToLocalString(dateFormat)
};
public Member? TryGetMember(string id) =>
Members.FirstOrDefault(m => m.Id == id);
public Member? TryGetMember(Snowflake id) => Members.FirstOrDefault(m => m.Id == id);
public Channel? TryGetChannel(string id) =>
Channels.FirstOrDefault(c => c.Id == id);
public Channel? TryGetChannel(Snowflake id) => Channels.FirstOrDefault(c => c.Id == id);
public Role? TryGetRole(string id) =>
Roles.FirstOrDefault(r => r.Id == id);
public Role? TryGetRole(Snowflake id) => Roles.FirstOrDefault(r => r.Id == id);
public Color? TryGetUserColor(string id)
public Color? TryGetUserColor(Snowflake id)
{
var member = TryGetMember(id);
var roles = member?.RoleIds.Join(Roles, i => i, r => r.Id, (_, role) => role);
@@ -1,6 +1,6 @@
using System;
using System.IO;
using System.IO;
using System.Text;
using DiscordChatExporter.Domain.Discord;
using DiscordChatExporter.Domain.Discord.Models;
using DiscordChatExporter.Domain.Internal;
@@ -22,9 +22,9 @@ namespace DiscordChatExporter.Domain.Exporting
public ExportFormat Format { get; }
public DateTimeOffset? After { get; }
public Snowflake? After { get; }
public DateTimeOffset? Before { get; }
public Snowflake? Before { get; }
public int? PartitionLimit { get; }
@@ -39,8 +39,8 @@ namespace DiscordChatExporter.Domain.Exporting
Channel channel,
string outputPath,
ExportFormat format,
DateTimeOffset? after,
DateTimeOffset? before,
Snowflake? after,
Snowflake? before,
int? partitionLimit,
bool shouldDownloadMedia,
bool shouldReuseMedia,
@@ -78,8 +78,8 @@ namespace DiscordChatExporter.Domain.Exporting
Channel channel,
string outputPath,
ExportFormat format,
DateTimeOffset? after = null,
DateTimeOffset? before = null)
Snowflake? after = null,
Snowflake? before = null)
{
// Output is a directory
if (Directory.Exists(outputPath) || string.IsNullOrWhiteSpace(Path.GetExtension(outputPath)))
@@ -96,8 +96,8 @@ namespace DiscordChatExporter.Domain.Exporting
Guild guild,
Channel channel,
ExportFormat format,
DateTimeOffset? after = null,
DateTimeOffset? before = null)
Snowflake? after = null,
Snowflake? before = null)
{
var buffer = new StringBuilder();
@@ -112,17 +112,17 @@ namespace DiscordChatExporter.Domain.Exporting
// Both 'after' and 'before' are set
if (after != null && before != null)
{
buffer.Append($"{after:yyyy-MM-dd} to {before:yyyy-MM-dd}");
buffer.Append($"{after?.ToDate():yyyy-MM-dd} to {before?.ToDate():yyyy-MM-dd}");
}
// Only 'after' is set
else if (after != null)
{
buffer.Append($"after {after:yyyy-MM-dd}");
buffer.Append($"after {after?.ToDate():yyyy-MM-dd}");
}
// Only 'before' is set
else
{
buffer.Append($"before {before:yyyy-MM-dd}");
buffer.Append($"before {before?.ToDate():yyyy-MM-dd}");
}
buffer.Append(")");
@@ -1,5 +1,6 @@
using System;
using System.Collections.Generic;
using System.Globalization;
using System.IO;
using System.Net.Http;
using System.Security.Cryptography;
@@ -8,6 +9,7 @@ using System.Text.RegularExpressions;
using System.Threading.Tasks;
using DiscordChatExporter.Domain.Internal;
using DiscordChatExporter.Domain.Internal.Extensions;
using DiscordChatExporter.Domain.Utilities;
namespace DiscordChatExporter.Domain.Exporting
{
@@ -18,8 +20,7 @@ namespace DiscordChatExporter.Domain.Exporting
private readonly bool _reuseMedia;
// URL -> Local file path
private readonly Dictionary<string, string> _pathCache =
new Dictionary<string, string>(StringComparer.Ordinal);
private readonly Dictionary<string, string> _pathCache = new(StringComparer.Ordinal);
public MediaDownloader(HttpClient httpClient, string workingDirPath, bool reuseMedia)
{
@@ -43,12 +44,31 @@ namespace DiscordChatExporter.Domain.Exporting
if (_reuseMedia && File.Exists(filePath))
return _pathCache[url] = filePath;
// Download it
Directory.CreateDirectory(_workingDirPath);
// This catches IOExceptions which is dangerous as we're working also with files
await Http.ExceptionPolicy.ExecuteAsync(async () =>
{
// This catches IOExceptions which is dangerous as we're working also with files
await _httpClient.DownloadAsync(url, filePath);
// Download the file
using var response = await _httpClient.GetAsync(url);
await using (var output = File.Create(filePath))
{
await response.Content.CopyToAsync(output);
}
// Try to set the file date according to the last-modified header
var lastModified = response.Content.Headers.TryGetValue("Last-Modified")?.Pipe(s =>
DateTimeOffset.TryParse(s, CultureInfo.InvariantCulture, DateTimeStyles.None, out var date)
? date
: (DateTimeOffset?) null
);
if (lastModified != null)
{
File.SetCreationTimeUtc(filePath, lastModified.Value.UtcDateTime);
File.SetLastWriteTimeUtc(filePath, lastModified.Value.UtcDateTime);
File.SetLastAccessTimeUtc(filePath, lastModified.Value.UtcDateTime);
}
});
return _pathCache[url] = filePath;
@@ -59,7 +59,7 @@ namespace DiscordChatExporter.Domain.Exporting.Writers
public override async ValueTask WriteMessageAsync(Message message)
{
// Author ID
await _writer.WriteAsync(CsvEncode(message.Author.Id));
await _writer.WriteAsync(CsvEncode(message.Author.Id.ToString()));
await _writer.WriteAsync(',');
// Author name
@@ -93,24 +93,20 @@ img {
filter: blur(44px);
}
.spoiler--hidden .spoiler-image:after {
content: "SPOILER";
color: #dcddde;
background-color: rgba(0, 0, 0, 0.6);
position: absolute;
left: 50%;
top: 50%;
transform: translate(-50%, -50%);
font-weight: 600;
/* padding: 0.5em 0.7em; */
padding: 100%;
/* This ruins those beutifully rounded buttons, but it's needed to prevent a FireFox bug with video and audio elemnts. */
/* The bug is that you can click trough the spoiler layer and play the video or audio, I could not identify the cause. */
/* I leave this here, in case someone is brave enough to venture in to madness that is undocumented browser behaviour. */
border-radius: 20px;
letter-spacing: 0.05em;
font-size: 0.9em;
}
.spoiler--hidden .spoiler-image:after {
content: "SPOILER";
color: #dcddde;
background-color: rgba(0, 0, 0, 0.6);
position: absolute;
left: 50%;
top: 50%;
transform: translate(-50%, -50%);
font-weight: 600;
padding: 100%;
border-radius: 20px;
letter-spacing: 0.05em;
font-size: 0.9em;
}
.spoiler--hidden:hover .spoiler-image:after {
color: #fff;
@@ -211,6 +207,58 @@ img {
grid-template-columns: auto 1fr;
}
.chatlog__reference-symbol {
grid-column: 1;
border-style: solid;
border-width: 2px 0 0 2px;
border-radius: 8px 0 0 0;
margin-left: 16px;
margin-top: 8px;
}
.chatlog__reference {
display: flex;
grid-column: 2;
margin-left: 1.2em;
margin-bottom: 0.25em;
font-size: 0.875em;
white-space: nowrap;
overflow: hidden;
text-overflow: ellipsis;
align-items: center;
}
.chatlog__reference-avatar {
border-radius: 50%;
height: 16px;
width: 16px;
margin-right: 0.25em;
}
.chatlog__reference-name {
margin-right: 0.25em;
font-weight: 600;
}
.chatlog__reference-link {
flex-grow: 1;
overflow: hidden;
text-overflow: ellipsis;
}
.chatlog__reference-link:hover {
text-decoration: none;
}
.chatlog__reference-content > * {
display: inline;
}
.chatlog__reference-edited-timestamp {
margin-left: 0.25em;
font-size: 0.8em;
}
.chatlog__author-avatar-container {
grid-column: 1;
width: 40px;
@@ -266,6 +314,53 @@ img {
border-radius: 3px;
}
.chatlog__attachment-container {
height: 40px;
width: 100%;
max-width: 520px;
padding: 10px;
border: 1px solid;
border-radius: 3px;
overflow: hidden;
}
.chatlog__attachment-icon {
float: left;
height: 100%;
margin-right: 10px;
}
.chatlog__attachment-icon > .a {
fill: #f4f5fb;
d: path("M50,935a25,25,0,0,1-25-25V50A25,25,0,0,1,50,25H519.6L695,201.32V910a25,25,0,0,1-25,25Z");
}
.chatlog__attachment-icon > .b {
fill: #7789c4;
d: path("M509.21,50,670,211.63V910H50V50H509.21M530,0H50A50,50,0,0,0,0,50V910a50,50,0,0,0,50,50H670a50,50,0,0,0,50-50h0V191Z");
}
.chatlog__attachment-icon > .c {
fill: #f4f5fb;
d: path("M530,215a25,25,0,0,1-25-25V50a25,25,0,0,1,16.23-23.41L693.41,198.77A25,25,0,0,1,670,215Z");
}
.chatlog__attachment-icon > .d {
fill: #7789c4;
d: path("M530,70.71,649.29,190H530V70.71M530,0a50,50,0,0,0-50,50V190a50,50,0,0,0,50,50H670a50,50,0,0,0,50-50Z");
}
.chatlog__attachment-filesize {
color: #72767d;
font-size: 12px;
}
.chatlog__attachment-filename {
overflow: hidden;
white-space: nowrap;
text-overflow: ellipsis;
}
.chatlog__embed {
display: flex;
margin-top: 0.3em;
@@ -46,6 +46,26 @@ a {
border-color: rgba(255, 255, 255, 0.1);
}
.chatlog__reference-symbol {
border-color: #4f545c;
}
.chatlog__reference {
color: #b5b6b8;
}
.chatlog__reference-link {
color: #b5b6b8;
}
.chatlog__reference-link:hover {
color: #ffffff;
}
.chatlog__reference-edited-timestamp {
color: rgba(255, 255, 255, 0.2);
}
.chatlog__author-name {
color: #ffffff;
}
@@ -62,6 +82,11 @@ a {
background-color: rgba(249, 168, 37, 0.05);
}
.chatlog__attachment-container {
background-color: #2f3136;
border-color: #292b2f;
}
.chatlog__edited-timestamp {
color: rgba(255, 255, 255, 0.2);
}
@@ -47,6 +47,26 @@ a {
border-color: #eceeef;
}
.chatlog__reference-symbol {
border-color: #c7ccd1;
}
.chatlog__reference {
color: #5f5f60;
}
.chatlog__reference-link {
color: #5f5f60;
}
.chatlog__reference-link:hover {
color: #2f3136;
}
.chatlog__reference-edited-timestamp {
color: #747f8d;
}
.chatlog__author-name {
font-weight: 600;
color: #2f3136;
@@ -64,6 +84,11 @@ a {
background-color: rgba(249, 168, 37, 0.05);
}
.chatlog__attachment-container {
background-color: #f2f3f5;
border-color: #ebedef;
}
.chatlog__edited-timestamp {
color: #747f8d;
}
@@ -14,10 +14,21 @@ namespace DiscordChatExporter.Domain.Exporting.Writers.Html
public IReadOnlyList<Message> Messages { get; }
public MessageGroup(User author, DateTimeOffset timestamp, IReadOnlyList<Message> messages)
public MessageReference? Reference { get; }
public Message? ReferencedMessage {get; }
public MessageGroup(
User author,
DateTimeOffset timestamp,
MessageReference? reference,
Message? referencedMessage,
IReadOnlyList<Message> messages)
{
Author = author;
Timestamp = timestamp;
Reference = reference;
ReferencedMessage = referencedMessage;
Messages = messages;
}
}
@@ -25,9 +36,10 @@ namespace DiscordChatExporter.Domain.Exporting.Writers.Html
internal partial class MessageGroup
{
public static bool CanJoin(Message message1, Message message2) =>
string.Equals(message1.Author.Id, message2.Author.Id, StringComparison.Ordinal) &&
message1.Author.Id == message2.Author.Id &&
string.Equals(message1.Author.FullName, message2.Author.FullName, StringComparison.Ordinal) &&
(message2.Timestamp - message1.Timestamp).Duration().TotalMinutes <= 7;
(message2.Timestamp - message1.Timestamp).Duration().TotalMinutes <= 7 &&
message2.Reference is null;
public static MessageGroup Join(IReadOnlyList<Message> messages)
{
@@ -36,6 +48,8 @@ namespace DiscordChatExporter.Domain.Exporting.Writers.Html
return new MessageGroup(
first.Author,
first.Timestamp,
first.Reference,
first.ReferencedMessage,
messages
);
}
@@ -1,7 +1,8 @@
@using System
@using System.Linq
@using System.Threading.Tasks
@inherits MiniRazor.MiniRazorTemplateBase<DiscordChatExporter.Domain.Exporting.Writers.Html.MessageGroupTemplateContext>
@namespace DiscordChatExporter.Domain.Exporting.Writers.Html
@inherits MiniRazor.TemplateBase<DiscordChatExporter.Domain.Exporting.Writers.Html.MessageGroupTemplateContext>
@{
string FormatDate(DateTimeOffset date) => Model.ExportContext.FormatDate(date);
@@ -13,15 +14,72 @@
ValueTask<string> ResolveUrlAsync(string url) => Model.ExportContext.ResolveMediaUrlAsync(url);
var userMember = Model.ExportContext.TryGetMember(Model.MessageGroup.Author.Id);
var userColor = Model.ExportContext.TryGetUserColor(Model.MessageGroup.Author.Id);
var userNick = Model.MessageGroup.Author.IsBot ? Model.MessageGroup.Author.Name : userMember?.Nick ?? Model.MessageGroup.Author.Name;
var userColorStyle = userColor != null
? $"color: rgb({userColor?.R},{userColor?.G},{userColor?.B})"
: null;
var userNick = Model.MessageGroup.Author.IsBot
? Model.MessageGroup.Author.Name
: userMember?.Nick ?? Model.MessageGroup.Author.Name;
var referencedUserMember = Model.MessageGroup.ReferencedMessage != null
? Model.ExportContext.TryGetMember(Model.MessageGroup.ReferencedMessage.Author.Id)
: null;
var referencedUserColor = Model.MessageGroup.ReferencedMessage != null
? Model.ExportContext.TryGetUserColor(Model.MessageGroup.ReferencedMessage.Author.Id)
: null;
var referencedUserColorStyle = referencedUserColor != null
? $"color: rgb({referencedUserColor?.R},{referencedUserColor?.G},{referencedUserColor?.B})"
: null;
var referencedUserNick = Model.MessageGroup.ReferencedMessage != null
? Model.MessageGroup.ReferencedMessage.Author.IsBot
? Model.MessageGroup.ReferencedMessage.Author.Name
: referencedUserMember?.Nick ?? Model.MessageGroup.ReferencedMessage.Author.Name
: null;
}
<div class="chatlog__message-group">
@if (Model.MessageGroup.Reference != null)
{
<div class="chatlog__reference-symbol">
</div>
<div class="chatlog__reference">
@if (Model.MessageGroup.ReferencedMessage != null)
{
<img class="chatlog__reference-avatar" src="@await ResolveUrlAsync(Model.MessageGroup.ReferencedMessage.Author.AvatarUrl)" alt="Avatar">
<span class="chatlog__reference-name" title="@Model.MessageGroup.ReferencedMessage.Author.FullName" style="@referencedUserColorStyle">@referencedUserNick</span>
<a class="chatlog__reference-link" href="#" onclick="scrollToMessage(event, '@Model.MessageGroup.ReferencedMessage.Id')">
<span class="chatlog__reference-content">
@if (!string.IsNullOrWhiteSpace(Model.MessageGroup.ReferencedMessage.Content))
{
@Raw(FormatMarkdown(Model.MessageGroup.ReferencedMessage.Content))
}
else
{
<em>Click to see original message</em>
}
</span>
@if (Model.MessageGroup.ReferencedMessage.EditedTimestamp != null)
{
<span class="chatlog__reference-edited-timestamp" title="@FormatDate(Model.MessageGroup.ReferencedMessage.EditedTimestamp.Value)">(edited)</span>
}
</a>
}
else
{
<span class="chatlog__reference-unknown">
Original message was deleted.
</span>
}
</div>
}
<div class="chatlog__author-avatar-container">
<img class="chatlog__author-avatar" src="@await ResolveUrlAsync(Model.MessageGroup.Author.AvatarUrl)" alt="Avatar">
</div>
@@ -79,9 +137,22 @@
}
else
{
<a href="@await ResolveUrlAsync(attachment.Url)">
@($"Attachment: {attachment.FileName} ({attachment.FileSize})")
</a>
<div class="chatlog__attachment-container">
<svg class="chatlog__attachment-icon" viewBox="0 0 720 960">
<path class="a"/>
<path class="b"/>
<path class="c"/>
<path class="d"/>
</svg>
<div class="chatlog__attachment-filename">
<a href="@await ResolveUrlAsync(attachment.Url)">
@attachment.FileName
</a>
</div>
<div class="chatlog__attachment-filesize">
@attachment.FileSize
</div>
</div>
}
</div>
</div>
@@ -1,4 +1,5 @@
@inherits MiniRazor.MiniRazorTemplateBase<DiscordChatExporter.Domain.Exporting.Writers.Html.LayoutTemplateContext>
@namespace DiscordChatExporter.Domain.Exporting.Writers.Html
@inherits MiniRazor.TemplateBase<DiscordChatExporter.Domain.Exporting.Writers.Html.LayoutTemplateContext>
</div>
@@ -7,4 +8,5 @@
</div>
</body>
</html>
@@ -1,7 +1,8 @@
@using System
@using System.Threading.Tasks
@using Tyrrrz.Extensions
@inherits MiniRazor.MiniRazorTemplateBase<DiscordChatExporter.Domain.Exporting.Writers.Html.LayoutTemplateContext>
@namespace DiscordChatExporter.Domain.Exporting.Writers.Html
@inherits MiniRazor.TemplateBase<DiscordChatExporter.Domain.Exporting.Writers.Html.LayoutTemplateContext>
@{
string FormatDate(DateTimeOffset date) => Model.ExportContext.FormatDate(date);
@@ -82,15 +83,15 @@
<div class="preamble__entry preamble__entry--small">
@if (Model.ExportContext.Request.After != null && Model.ExportContext.Request.Before != null)
{
@($"Between {FormatDate(Model.ExportContext.Request.After.Value)} and {FormatDate(Model.ExportContext.Request.Before.Value)}")
@($"Between {FormatDate(Model.ExportContext.Request.After.Value.ToDate())} and {FormatDate(Model.ExportContext.Request.Before.Value.ToDate())}")
}
else if (Model.ExportContext.Request.After != null)
{
@($"After {FormatDate(Model.ExportContext.Request.After.Value)}")
@($"After {FormatDate(Model.ExportContext.Request.After.Value.ToDate())}")
}
else if (Model.ExportContext.Request.Before != null)
{
@($"Before {FormatDate(Model.ExportContext.Request.Before.Value)}")
@($"Before {FormatDate(Model.ExportContext.Request.Before.Value.ToDate())}")
}
</div>
}
@@ -1,65 +0,0 @@
using System.Runtime.CompilerServices;
using System.Threading.Tasks;
using MiniRazor;
using Tyrrrz.Extensions;
[assembly: InternalsVisibleTo(DiscordChatExporter.Domain.Exporting.Writers.Html.TemplateBundle.PreambleTemplateAssemblyName)]
[assembly: InternalsVisibleTo(DiscordChatExporter.Domain.Exporting.Writers.Html.TemplateBundle.MessageGroupTemplateAssemblyName)]
[assembly: InternalsVisibleTo(DiscordChatExporter.Domain.Exporting.Writers.Html.TemplateBundle.PostambleTemplateAssemblyName)]
namespace DiscordChatExporter.Domain.Exporting.Writers.Html
{
internal partial class TemplateBundle
{
public const string PreambleTemplateAssemblyName = "RazorAssembly_Preamble";
public const string MessageGroupTemplateAssemblyName = "RazorAssembly_MessageGroup";
public const string PostambleTemplateAssemblyName = "RazorAssembly_Postamble";
public MiniRazorTemplateDescriptor PreambleTemplate { get; }
public MiniRazorTemplateDescriptor MessageGroupTemplate { get; }
public MiniRazorTemplateDescriptor PostambleTemplate { get; }
public TemplateBundle(
MiniRazorTemplateDescriptor preambleTemplate,
MiniRazorTemplateDescriptor messageGroupTemplate,
MiniRazorTemplateDescriptor postambleTemplate)
{
PreambleTemplate = preambleTemplate;
MessageGroupTemplate = messageGroupTemplate;
PostambleTemplate = postambleTemplate;
}
}
internal partial class TemplateBundle
{
private static TemplateBundle? _lastBundle;
// This is very CPU-heavy
private static async ValueTask<TemplateBundle> CompileAsync() => await Task.Run(() =>
{
var ns = typeof(TemplateBundle).Namespace!;
var preambleTemplateSource = typeof(HtmlMessageWriter).Assembly
.GetManifestResourceString($"{ns}.PreambleTemplate.cshtml");
var messageGroupTemplateSource = typeof(HtmlMessageWriter).Assembly
.GetManifestResourceString($"{ns}.MessageGroupTemplate.cshtml");
var postambleTemplateSource = typeof(HtmlMessageWriter).Assembly
.GetManifestResourceString($"{ns}.PostambleTemplate.cshtml");
var engine = new MiniRazorTemplateEngine();
var preambleTemplate = engine.Compile(preambleTemplateSource, PreambleTemplateAssemblyName, ns);
var messageGroupTemplate = engine.Compile(messageGroupTemplateSource, MessageGroupTemplateAssemblyName, ns);
var postambleTemplate = engine.Compile(postambleTemplateSource, PostambleTemplateAssemblyName, ns);
return new TemplateBundle(preambleTemplate, messageGroupTemplate, postambleTemplate);
});
public static async ValueTask<TemplateBundle> ResolveAsync() =>
_lastBundle ??= await CompileAsync();
}
}
@@ -12,7 +12,7 @@ namespace DiscordChatExporter.Domain.Exporting.Writers
private readonly TextWriter _writer;
private readonly string _themeName;
private readonly List<Message> _messageGroupBuffer = new List<Message>();
private readonly List<Message> _messageGroupBuffer = new();
private long _messageCount;
@@ -26,20 +26,18 @@ namespace DiscordChatExporter.Domain.Exporting.Writers
public override async ValueTask WritePreambleAsync()
{
var templateContext = new LayoutTemplateContext(Context, _themeName, _messageCount);
var templateBundle = await TemplateBundle.ResolveAsync();
await _writer.WriteLineAsync(
await templateBundle.PreambleTemplate.RenderAsync(templateContext)
await PreambleTemplate.RenderAsync(templateContext)
);
}
private async ValueTask WriteMessageGroupAsync(MessageGroup messageGroup)
{
var templateContext = new MessageGroupTemplateContext(Context, messageGroup);
var templateBundle = await TemplateBundle.ResolveAsync();
await _writer.WriteLineAsync(
await templateBundle.MessageGroupTemplate.RenderAsync(templateContext)
await MessageGroupTemplate.RenderAsync(templateContext)
);
}
@@ -70,10 +68,9 @@ namespace DiscordChatExporter.Domain.Exporting.Writers
await WriteMessageGroupAsync(MessageGroup.Join(_messageGroupBuffer));
var templateContext = new LayoutTemplateContext(Context, _themeName, _messageCount);
var templateBundle = await TemplateBundle.ResolveAsync();
await _writer.WriteLineAsync(
await templateBundle.PostambleTemplate.RenderAsync(templateContext)
await PostambleTemplate.RenderAsync(templateContext)
);
}
@@ -1,10 +1,11 @@
using System.IO;
using System.Text.Json;
using System.Threading.Tasks;
using System.Text.Encodings.Web;
using DiscordChatExporter.Domain.Discord.Models;
using DiscordChatExporter.Domain.Exporting.Writers.MarkdownVisitors;
using DiscordChatExporter.Domain.Internal.Extensions;
using DiscordChatExporter.Domain.Markdown.Ast;
using JsonExtensions.Writing;
namespace DiscordChatExporter.Domain.Exporting.Writers
{
@@ -19,6 +20,7 @@ namespace DiscordChatExporter.Domain.Exporting.Writers
{
_writer = new Utf8JsonWriter(stream, new JsonWriterOptions
{
Encoder = JavaScriptEncoder.UnsafeRelaxedJsonEscaping,
Indented = true
});
}
@@ -30,7 +32,7 @@ namespace DiscordChatExporter.Domain.Exporting.Writers
{
_writer.WriteStartObject();
_writer.WriteString("id", attachment.Id);
_writer.WriteString("id", attachment.Id.ToString());
_writer.WriteString("url", await Context.ResolveMediaUrlAsync(attachment.Url));
_writer.WriteString("fileName", attachment.FileName);
_writer.WriteNumber("fileSizeBytes", attachment.FileSize.TotalBytes);
@@ -164,7 +166,7 @@ namespace DiscordChatExporter.Domain.Exporting.Writers
{
_writer.WriteStartObject();
_writer.WriteString("id", mentionedUser.Id);
_writer.WriteString("id", mentionedUser.Id.ToString());
_writer.WriteString("name", mentionedUser.Name);
_writer.WriteNumber("discriminator", mentionedUser.Discriminator);
_writer.WriteString("nickname", Context.TryGetMember(mentionedUser.Id)?.Nick ?? mentionedUser.Name);
@@ -181,14 +183,14 @@ namespace DiscordChatExporter.Domain.Exporting.Writers
// Guild
_writer.WriteStartObject("guild");
_writer.WriteString("id", Context.Request.Guild.Id);
_writer.WriteString("id", Context.Request.Guild.Id.ToString());
_writer.WriteString("name", Context.Request.Guild.Name);
_writer.WriteString("iconUrl", await Context.ResolveMediaUrlAsync(Context.Request.Guild.IconUrl));
_writer.WriteEndObject();
// Channel
_writer.WriteStartObject("channel");
_writer.WriteString("id", Context.Request.Channel.Id);
_writer.WriteString("id", Context.Request.Channel.Id.ToString());
_writer.WriteString("type", Context.Request.Channel.Type.ToString());
_writer.WriteString("category", Context.Request.Channel.Category);
_writer.WriteString("name", Context.Request.Channel.Name);
@@ -197,8 +199,8 @@ namespace DiscordChatExporter.Domain.Exporting.Writers
// Date range
_writer.WriteStartObject("dateRange");
_writer.WriteString("after", Context.Request.After);
_writer.WriteString("before", Context.Request.Before);
_writer.WriteString("after", Context.Request.After?.ToDate());
_writer.WriteString("before", Context.Request.Before?.ToDate());
_writer.WriteEndObject();
// Message array (start)
@@ -211,7 +213,7 @@ namespace DiscordChatExporter.Domain.Exporting.Writers
_writer.WriteStartObject();
// Metadata
_writer.WriteString("id", message.Id);
_writer.WriteString("id", message.Id.ToString());
_writer.WriteString("type", message.Type.ToString());
_writer.WriteString("timestamp", message.Timestamp);
_writer.WriteString("timestampEdited", message.EditedTimestamp);
@@ -223,7 +225,7 @@ namespace DiscordChatExporter.Domain.Exporting.Writers
// Author
_writer.WriteStartObject("author");
_writer.WriteString("id", message.Author.Id);
_writer.WriteString("id", message.Author.Id.ToString());
_writer.WriteString("name", message.Author.Name);
_writer.WriteString("discriminator", $"{message.Author.Discriminator:0000}");
_writer.WriteBoolean("isBot", message.Author.IsBot);
@@ -256,11 +258,22 @@ namespace DiscordChatExporter.Domain.Exporting.Writers
// Mentions
_writer.WriteStartArray("mentions");
foreach (var mention in message.MentionedUsers)
await WriteMentionAsync(mention);
_writer.WriteEndArray();
// Message reference
if (message.Reference != null)
{
_writer.WriteStartObject("reference");
_writer.WriteString("messageId", message.Reference.MessageId?.ToString());
_writer.WriteString("channelId", message.Reference.ChannelId?.ToString());
_writer.WriteString("guildId", message.Reference.GuildId?.ToString());
_writer.WriteEndObject();
}
_writer.WriteEndObject();
await _writer.FlushAsync();
@@ -3,6 +3,7 @@ using System.Linq;
using System.Net;
using System.Text;
using System.Text.RegularExpressions;
using DiscordChatExporter.Domain.Discord;
using DiscordChatExporter.Domain.Discord.Models;
using DiscordChatExporter.Domain.Markdown;
using DiscordChatExporter.Domain.Markdown.Ast;
@@ -84,7 +85,7 @@ namespace DiscordChatExporter.Domain.Exporting.Writers.MarkdownVisitors
}
else if (mention.Type == MentionType.User)
{
var member = _context.TryGetMember(mention.Id);
var member = _context.TryGetMember(Snowflake.Parse(mention.Id));
var fullName = member?.User.FullName ?? "Unknown";
var nick = member?.Nick ?? "Unknown";
@@ -95,7 +96,7 @@ namespace DiscordChatExporter.Domain.Exporting.Writers.MarkdownVisitors
}
else if (mention.Type == MentionType.Channel)
{
var channel = _context.TryGetChannel(mention.Id);
var channel = _context.TryGetChannel(Snowflake.Parse(mention.Id));
var name = channel?.Name ?? "deleted-channel";
_buffer
@@ -105,7 +106,7 @@ namespace DiscordChatExporter.Domain.Exporting.Writers.MarkdownVisitors
}
else if (mention.Type == MentionType.Role)
{
var role = _context.TryGetRole(mention.Id);
var role = _context.TryGetRole(Snowflake.Parse(mention.Id));
var name = role?.Name ?? "deleted-role";
var color = role?.Color;
@@ -1,4 +1,5 @@
using System.Text;
using DiscordChatExporter.Domain.Discord;
using DiscordChatExporter.Domain.Markdown;
using DiscordChatExporter.Domain.Markdown.Ast;
@@ -29,21 +30,21 @@ namespace DiscordChatExporter.Domain.Exporting.Writers.MarkdownVisitors
}
else if (mention.Type == MentionType.User)
{
var member = _context.TryGetMember(mention.Id);
var member = _context.TryGetMember(Snowflake.Parse(mention.Id));
var name = member?.User.Name ?? "Unknown";
_buffer.Append($"@{name}");
}
else if (mention.Type == MentionType.Channel)
{
var channel = _context.TryGetChannel(mention.Id);
var channel = _context.TryGetChannel(Snowflake.Parse(mention.Id));
var name = channel?.Name ?? "deleted-channel";
_buffer.Append($"#{name}");
}
else if (mention.Type == MentionType.Role)
{
var role = _context.TryGetRole(mention.Id);
var role = _context.TryGetRole(Snowflake.Parse(mention.Id));
var name = role?.Name ?? "deleted-role";
_buffer.Append($"@{name}");
@@ -119,10 +119,10 @@ namespace DiscordChatExporter.Domain.Exporting.Writers
await _writer.WriteLineAsync($"Topic: {Context.Request.Channel.Topic}");
if (Context.Request.After != null)
await _writer.WriteLineAsync($"After: {Context.FormatDate(Context.Request.After.Value)}");
await _writer.WriteLineAsync($"After: {Context.FormatDate(Context.Request.After.Value.ToDate())}");
if (Context.Request.Before != null)
await _writer.WriteLineAsync($"Before: {Context.FormatDate(Context.Request.Before.Value)}");
await _writer.WriteLineAsync($"Before: {Context.FormatDate(Context.Request.Before.Value.ToDate())}");
await _writer.WriteLineAsync('='.Repeat(62));
await _writer.WriteLineAsync();
@@ -5,12 +5,6 @@ namespace DiscordChatExporter.Domain.Internal.Extensions
{
internal static class DateExtensions
{
public static string ToSnowflake(this DateTimeOffset dateTime)
{
var value = ((ulong) dateTime.ToUnixTimeMilliseconds() - 1420070400000UL) << 22;
return value.ToString();
}
public static string ToLocalString(this DateTimeOffset dateTime, string format) =>
dateTime.ToLocalTime().ToString(format, CultureInfo.InvariantCulture);
}
@@ -4,8 +4,6 @@ namespace DiscordChatExporter.Domain.Internal.Extensions
{
internal static class GenericExtensions
{
public static TOut Pipe<TIn, TOut>(this TIn input, Func<TIn, TOut> transform) => transform(input);
public static T? NullIf<T>(this T value, Func<T, bool> predicate) where T : struct =>
!predicate(value)
? value
@@ -1,27 +0,0 @@
using System.IO;
using System.Net.Http;
using System.Text.Json;
using System.Threading.Tasks;
namespace DiscordChatExporter.Domain.Internal.Extensions
{
internal static class HttpClientExtensions
{
public static async ValueTask DownloadAsync(this HttpClient httpClient, string uri, string outputFilePath)
{
await using var input = await httpClient.GetStreamAsync(uri);
var output = File.Create(outputFilePath);
await input.CopyToAsync(output);
await output.DisposeAsync();
}
public static async ValueTask<JsonElement> ReadAsJsonAsync(this HttpContent content)
{
await using var stream = await content.ReadAsStreamAsync();
using var doc = await JsonDocument.ParseAsync(stream);
return doc.RootElement.Clone();
}
}
}
@@ -0,0 +1,12 @@
using System.Net.Http.Headers;
namespace DiscordChatExporter.Domain.Internal.Extensions
{
internal static class HttpExtensions
{
public static string? TryGetValue(this HttpContentHeaders headers, string name) =>
headers.TryGetValues(name, out var values)
? string.Concat(values)
: null;
}
}
@@ -1,12 +0,0 @@
using System.Text.Json;
namespace DiscordChatExporter.Domain.Internal.Extensions
{
internal static class JsonElementExtensions
{
public static JsonElement? GetPropertyOrNull(this JsonElement element, string propertyName) =>
element.TryGetProperty(propertyName, out var result) && result.ValueKind != JsonValueKind.Null
? result
: (JsonElement?) null;
}
}
@@ -1,28 +0,0 @@
using System;
using System.Text.Json;
namespace DiscordChatExporter.Domain.Internal.Extensions
{
internal static class Utf8JsonWriterExtensions
{
public static void WriteString(this Utf8JsonWriter writer, string propertyName, DateTimeOffset? value)
{
writer.WritePropertyName(propertyName);
if (value != null)
writer.WriteStringValue(value.Value);
else
writer.WriteNullValue();
}
public static void WriteNumber(this Utf8JsonWriter writer, string propertyName, int? value)
{
writer.WritePropertyName(propertyName);
if (value != null)
writer.WriteNumberValue(value.Value);
else
writer.WriteNullValue();
}
}
}
+4 -4
View File
@@ -11,7 +11,7 @@ namespace DiscordChatExporter.Domain.Internal
{
internal static class Http
{
public static HttpClient Client { get; } = new HttpClient();
public static HttpClient Client { get; } = new();
public static IAsyncPolicy<HttpResponseMessage> ResponsePolicy { get; } =
Policy
@@ -21,10 +21,10 @@ namespace DiscordChatExporter.Domain.Internal
.OrResult(m => m.StatusCode == HttpStatusCode.RequestTimeout)
.OrResult(m => m.StatusCode >= HttpStatusCode.InternalServerError)
.WaitAndRetryAsync(8,
(i, result, ctx) =>
(i, result, _) =>
{
// If rate-limited, use retry-after as a guide
if (result.Result.StatusCode == HttpStatusCode.TooManyRequests)
if (result.Result?.StatusCode == HttpStatusCode.TooManyRequests)
{
// Only start respecting retry-after after a few attempts.
// The reason is that Discord often sends unreasonable (20+ minutes) retry-after
@@ -39,7 +39,7 @@ namespace DiscordChatExporter.Domain.Internal
return TimeSpan.FromSeconds(Math.Pow(2, i) + 1);
},
(response, timespan, retryCount, context) => Task.CompletedTask);
(_, _, _, _) => Task.CompletedTask);
private static HttpStatusCode? TryGetStatusCodeFromException(HttpRequestException ex)
{
@@ -10,8 +10,7 @@ namespace DiscordChatExporter.Domain.Internal
{
private string _path = "";
private readonly Dictionary<string, string?> _queryParameters =
new Dictionary<string, string?>(StringComparer.OrdinalIgnoreCase);
private readonly Dictionary<string, string?> _queryParameters = new(StringComparer.OrdinalIgnoreCase);
public UrlBuilder SetPath(string path)
{
@@ -9,74 +9,92 @@ namespace DiscordChatExporter.Domain.Markdown
// The following parsing logic is meant to replicate Discord's markdown grammar as close as possible
internal static partial class MarkdownParser
{
private const RegexOptions DefaultRegexOptions = RegexOptions.Compiled | RegexOptions.CultureInvariant | RegexOptions.Multiline;
private const RegexOptions DefaultRegexOptions =
RegexOptions.Compiled |
RegexOptions.CultureInvariant |
RegexOptions.Multiline;
/* Formatting */
// Capture any character until the earliest double asterisk not followed by an asterisk
private static readonly IMatcher<MarkdownNode> BoldFormattedNodeMatcher = new RegexMatcher<MarkdownNode>(
new Regex("\\*\\*(.+?)\\*\\*(?!\\*)", DefaultRegexOptions | RegexOptions.Singleline),
(p, m) => new FormattedNode(TextFormatting.Bold, Parse(p.Slice(m.Groups[1]))));
(p, m) => new FormattedNode(TextFormatting.Bold, Parse(p.Slice(m.Groups[1])))
);
// Capture any character until the earliest single asterisk not preceded or followed by an asterisk
// Opening asterisk must not be followed by whitespace
// Closing asterisk must not be preceded by whitespace
private static readonly IMatcher<MarkdownNode> ItalicFormattedNodeMatcher = new RegexMatcher<MarkdownNode>(
new Regex("\\*(?!\\s)(.+?)(?<!\\s|\\*)\\*(?!\\*)", DefaultRegexOptions | RegexOptions.Singleline),
(p, m) => new FormattedNode(TextFormatting.Italic, Parse(p.Slice(m.Groups[1]))));
(p, m) => new FormattedNode(TextFormatting.Italic, Parse(p.Slice(m.Groups[1])))
);
// Capture any character until the earliest triple asterisk not followed by an asterisk
private static readonly IMatcher<MarkdownNode> ItalicBoldFormattedNodeMatcher = new RegexMatcher<MarkdownNode>(
new Regex("\\*(\\*\\*.+?\\*\\*)\\*(?!\\*)", DefaultRegexOptions | RegexOptions.Singleline),
(p, m) => new FormattedNode(TextFormatting.Italic, Parse(p.Slice(m.Groups[1]), BoldFormattedNodeMatcher)));
(p, m) => new FormattedNode(TextFormatting.Italic, Parse(p.Slice(m.Groups[1]), BoldFormattedNodeMatcher))
);
// Capture any character except underscore until an underscore
// Closing underscore must not be followed by a word character
private static readonly IMatcher<MarkdownNode> ItalicAltFormattedNodeMatcher = new RegexMatcher<MarkdownNode>(
new Regex("_([^_]+)_(?!\\w)", DefaultRegexOptions | RegexOptions.Singleline),
(p, m) => new FormattedNode(TextFormatting.Italic, Parse(p.Slice(m.Groups[1]))));
(p, m) => new FormattedNode(TextFormatting.Italic, Parse(p.Slice(m.Groups[1])))
);
// Capture any character until the earliest double underscore not followed by an underscore
private static readonly IMatcher<MarkdownNode> UnderlineFormattedNodeMatcher = new RegexMatcher<MarkdownNode>(
new Regex("__(.+?)__(?!_)", DefaultRegexOptions | RegexOptions.Singleline),
(p, m) => new FormattedNode(TextFormatting.Underline, Parse(p.Slice(m.Groups[1]))));
(p, m) => new FormattedNode(TextFormatting.Underline, Parse(p.Slice(m.Groups[1])))
);
// Capture any character until the earliest triple underscore not followed by an underscore
private static readonly IMatcher<MarkdownNode> ItalicUnderlineFormattedNodeMatcher = new RegexMatcher<MarkdownNode>(
new Regex("_(__.+?__)_(?!_)", DefaultRegexOptions | RegexOptions.Singleline),
(p, m) => new FormattedNode(TextFormatting.Italic, Parse(p.Slice(m.Groups[1]), UnderlineFormattedNodeMatcher)));
private static readonly IMatcher<MarkdownNode> ItalicUnderlineFormattedNodeMatcher =
new RegexMatcher<MarkdownNode>(
new Regex("_(__.+?__)_(?!_)", DefaultRegexOptions | RegexOptions.Singleline),
(p, m) => new FormattedNode(TextFormatting.Italic,
Parse(p.Slice(m.Groups[1]), UnderlineFormattedNodeMatcher))
);
// Capture any character until the earliest double tilde
private static readonly IMatcher<MarkdownNode> StrikethroughFormattedNodeMatcher = new RegexMatcher<MarkdownNode>(
new Regex("~~(.+?)~~", DefaultRegexOptions | RegexOptions.Singleline),
(p, m) => new FormattedNode(TextFormatting.Strikethrough, Parse(p.Slice(m.Groups[1]))));
private static readonly IMatcher<MarkdownNode> StrikethroughFormattedNodeMatcher =
new RegexMatcher<MarkdownNode>(
new Regex("~~(.+?)~~", DefaultRegexOptions | RegexOptions.Singleline),
(p, m) => new FormattedNode(TextFormatting.Strikethrough, Parse(p.Slice(m.Groups[1])))
);
// Capture any character until the earliest double pipe
private static readonly IMatcher<MarkdownNode> SpoilerFormattedNodeMatcher = new RegexMatcher<MarkdownNode>(
new Regex("\\|\\|(.+?)\\|\\|", DefaultRegexOptions | RegexOptions.Singleline),
(p, m) => new FormattedNode(TextFormatting.Spoiler, Parse(p.Slice(m.Groups[1]))));
(p, m) => new FormattedNode(TextFormatting.Spoiler, Parse(p.Slice(m.Groups[1])))
);
// Capture any character until the end of the line
// Opening 'greater than' character must be followed by whitespace
private static readonly IMatcher<MarkdownNode> SingleLineQuoteNodeMatcher = new RegexMatcher<MarkdownNode>(
new Regex("^>\\s(.+\n?)", DefaultRegexOptions),
(p, m) => new FormattedNode(TextFormatting.Quote, Parse(p.Slice(m.Groups[1]))));
(p, m) => new FormattedNode(TextFormatting.Quote, Parse(p.Slice(m.Groups[1])))
);
// Repeatedly capture any character until the end of the line
// This one is tricky as it ends up producing multiple separate captures which need to be joined
private static readonly IMatcher<MarkdownNode> RepeatedSingleLineQuoteNodeMatcher = new RegexMatcher<MarkdownNode>(
new Regex("(?:^>\\s(.+\n?)){2,}", DefaultRegexOptions),
(p, m) =>
{
var content = string.Concat(m.Groups[1].Captures.Select(c => c.Value));
return new FormattedNode(TextFormatting.Quote, Parse(content));
});
private static readonly IMatcher<MarkdownNode> RepeatedSingleLineQuoteNodeMatcher =
new RegexMatcher<MarkdownNode>(
new Regex("(?:^>\\s(.+\n?)){2,}", DefaultRegexOptions),
(_, m) =>
{
var content = string.Concat(m.Groups[1].Captures.Select(c => c.Value));
return new FormattedNode(TextFormatting.Quote, Parse(content));
}
);
// Capture any character until the end of the input
// Opening 'greater than' characters must be followed by whitespace
private static readonly IMatcher<MarkdownNode> MultiLineQuoteNodeMatcher = new RegexMatcher<MarkdownNode>(
new Regex("^>>>\\s(.+)", DefaultRegexOptions | RegexOptions.Singleline),
(p, m) => new FormattedNode(TextFormatting.Quote, Parse(p.Slice(m.Groups[1]))));
(p, m) => new FormattedNode(TextFormatting.Quote, Parse(p.Slice(m.Groups[1])))
);
/* Code blocks */
@@ -85,41 +103,48 @@ namespace DiscordChatExporter.Domain.Markdown
// There can be either one or two backticks, but equal number on both sides
private static readonly IMatcher<MarkdownNode> InlineCodeBlockNodeMatcher = new RegexMatcher<MarkdownNode>(
new Regex("(`{1,2})([^`]+)\\1", DefaultRegexOptions | RegexOptions.Singleline),
m => new InlineCodeBlockNode(m.Groups[2].Value.Trim('\r', '\n')));
m => new InlineCodeBlockNode(m.Groups[2].Value.Trim('\r', '\n'))
);
// Capture language identifier and then any character until the earliest triple backtick
// Language identifier is one word immediately after opening backticks, followed immediately by newline
// Blank lines at the beginning and end of content are trimmed
private static readonly IMatcher<MarkdownNode> MultiLineCodeBlockNodeMatcher = new RegexMatcher<MarkdownNode>(
new Regex("```(?:(\\w*)\\n)?(.+?)```", DefaultRegexOptions | RegexOptions.Singleline),
m => new MultiLineCodeBlockNode(m.Groups[1].Value, m.Groups[2].Value.Trim('\r', '\n')));
m => new MultiLineCodeBlockNode(m.Groups[1].Value, m.Groups[2].Value.Trim('\r', '\n'))
);
/* Mentions */
// Capture @everyone
private static readonly IMatcher<MarkdownNode> EveryoneMentionNodeMatcher = new StringMatcher<MarkdownNode>(
"@everyone",
p => new MentionNode("everyone", MentionType.Meta));
_ => new MentionNode("everyone", MentionType.Meta)
);
// Capture @here
private static readonly IMatcher<MarkdownNode> HereMentionNodeMatcher = new StringMatcher<MarkdownNode>(
"@here",
p => new MentionNode("here", MentionType.Meta));
_ => new MentionNode("here", MentionType.Meta)
);
// Capture <@123456> or <@!123456>
private static readonly IMatcher<MarkdownNode> UserMentionNodeMatcher = new RegexMatcher<MarkdownNode>(
new Regex("<@!?(\\d+)>", DefaultRegexOptions),
m => new MentionNode(m.Groups[1].Value, MentionType.User));
m => new MentionNode(m.Groups[1].Value, MentionType.User)
);
// Capture <#123456>
private static readonly IMatcher<MarkdownNode> ChannelMentionNodeMatcher = new RegexMatcher<MarkdownNode>(
new Regex("<#(\\d+)>", DefaultRegexOptions),
m => new MentionNode(m.Groups[1].Value, MentionType.Channel));
m => new MentionNode(m.Groups[1].Value, MentionType.Channel)
);
// Capture <@&123456>
private static readonly IMatcher<MarkdownNode> RoleMentionNodeMatcher = new RegexMatcher<MarkdownNode>(
new Regex("<@&(\\d+)>", DefaultRegexOptions),
m => new MentionNode(m.Groups[1].Value, MentionType.Role));
m => new MentionNode(m.Groups[1].Value, MentionType.Role)
);
/* Emojis */
@@ -129,30 +154,36 @@ namespace DiscordChatExporter.Domain.Markdown
// ... or digit followed by enclosing mark
// (this does not match all emojis in Discord but it's reasonably accurate enough)
private static readonly IMatcher<MarkdownNode> StandardEmojiNodeMatcher = new RegexMatcher<MarkdownNode>(
new Regex("((?:[\\uD83C][\\uDDE6-\\uDDFF]){2}|[\\u2600-\\u26FF]|\\p{Cs}{2}|\\d\\p{Me})", DefaultRegexOptions),
m => new EmojiNode(m.Groups[1].Value));
new Regex("((?:[\\uD83C][\\uDDE6-\\uDDFF]){2}|[\\u2600-\\u26FF]|\\p{Cs}{2}|\\d\\p{Me})",
DefaultRegexOptions),
m => new EmojiNode(m.Groups[1].Value)
);
// Capture <:lul:123456> or <a:lul:123456>
private static readonly IMatcher<MarkdownNode> CustomEmojiNodeMatcher = new RegexMatcher<MarkdownNode>(
new Regex("<(a)?:(.+?):(\\d+?)>", DefaultRegexOptions),
m => new EmojiNode(m.Groups[3].Value, m.Groups[2].Value, !string.IsNullOrWhiteSpace(m.Groups[1].Value)));
m => new EmojiNode(m.Groups[3].Value, m.Groups[2].Value, !string.IsNullOrWhiteSpace(m.Groups[1].Value))
);
/* Links */
// Capture [title](link)
private static readonly IMatcher<MarkdownNode> TitledLinkNodeMatcher = new RegexMatcher<MarkdownNode>(
new Regex("\\[(.+?)\\]\\((.+?)\\)", DefaultRegexOptions),
m => new LinkNode(m.Groups[2].Value, m.Groups[1].Value));
m => new LinkNode(m.Groups[2].Value, m.Groups[1].Value)
);
// Capture any non-whitespace character after http:// or https:// until the last punctuation character or whitespace
private static readonly IMatcher<MarkdownNode> AutoLinkNodeMatcher = new RegexMatcher<MarkdownNode>(
new Regex("(https?://\\S*[^\\.,:;\"\'\\s])", DefaultRegexOptions),
m => new LinkNode(m.Groups[1].Value));
m => new LinkNode(m.Groups[1].Value)
);
// Same as auto link but also surrounded by angular brackets
private static readonly IMatcher<MarkdownNode> HiddenLinkNodeMatcher = new RegexMatcher<MarkdownNode>(
new Regex("<(https?://\\S*[^\\.,:;\"\'\\s])>", DefaultRegexOptions),
m => new LinkNode(m.Groups[1].Value));
m => new LinkNode(m.Groups[1].Value)
);
/* Text */
@@ -160,25 +191,29 @@ namespace DiscordChatExporter.Domain.Markdown
// This escapes it from matching for formatting
private static readonly IMatcher<MarkdownNode> ShrugTextNodeMatcher = new StringMatcher<MarkdownNode>(
@"¯\_(ツ)_/¯",
p => new TextNode(p.ToString()));
p => new TextNode(p.ToString())
);
// Capture some specific emojis that don't get rendered
// This escapes it from matching for emoji
private static readonly IMatcher<MarkdownNode> IgnoredEmojiTextNodeMatcher = new RegexMatcher<MarkdownNode>(
new Regex("(\\u26A7|\\u2640|\\u2642|\\u2695|\\u267E|\\u00A9|\\u00AE|\\u2122)", DefaultRegexOptions),
m => new TextNode(m.Groups[1].Value));
m => new TextNode(m.Groups[1].Value)
);
// Capture any "symbol/other" character or surrogate pair preceded by a backslash
// This escapes it from matching for emoji
private static readonly IMatcher<MarkdownNode> EscapedSymbolTextNodeMatcher = new RegexMatcher<MarkdownNode>(
new Regex("\\\\(\\p{So}|\\p{Cs}{2})", DefaultRegexOptions),
m => new TextNode(m.Groups[1].Value));
m => new TextNode(m.Groups[1].Value)
);
// Capture any non-whitespace, non latin alphanumeric character preceded by a backslash
// This escapes it from matching for formatting or other tokens
private static readonly IMatcher<MarkdownNode> EscapedCharacterTextNodeMatcher = new RegexMatcher<MarkdownNode>(
new Regex("\\\\([^a-zA-Z0-9\\s])", DefaultRegexOptions),
m => new TextNode(m.Groups[1].Value));
m => new TextNode(m.Groups[1].Value)
);
// Combine all matchers into one
// Matchers that have similar patterns are ordered from most specific to least specific
@@ -25,7 +25,7 @@ namespace DiscordChatExporter.Domain.Markdown.Matching
{
}
public StringPart Slice(int newStartIndex, int newLength) => new StringPart(Target, newStartIndex, newLength);
public StringPart Slice(int newStartIndex, int newLength) => new(Target, newStartIndex, newLength);
public StringPart Slice(int newStartIndex) => Slice(newStartIndex, EndIndex - newStartIndex);
@@ -0,0 +1,9 @@
using System;
namespace DiscordChatExporter.Domain.Utilities
{
public static class GeneralExtensions
{
public static TOut Pipe<TIn, TOut>(this TIn input, Func<TIn, TOut> transform) => transform(input);
}
}
@@ -7,7 +7,7 @@ namespace DiscordChatExporter.Gui.Converters
[ValueConversion(typeof(DateTimeOffset?), typeof(DateTime?))]
public class DateTimeOffsetToDateTimeConverter : IValueConverter
{
public static DateTimeOffsetToDateTimeConverter Instance { get; } = new DateTimeOffsetToDateTimeConverter();
public static DateTimeOffsetToDateTimeConverter Instance { get; } = new();
public object? Convert(object value, Type targetType, object parameter, CultureInfo culture)
{
@@ -8,7 +8,7 @@ namespace DiscordChatExporter.Gui.Converters
[ValueConversion(typeof(ExportFormat), typeof(string))]
public class ExportFormatToStringConverter : IValueConverter
{
public static ExportFormatToStringConverter Instance { get; } = new ExportFormatToStringConverter();
public static ExportFormatToStringConverter Instance { get; } = new();
public object? Convert(object value, Type targetType, object parameter, CultureInfo culture)
{
@@ -7,7 +7,7 @@ namespace DiscordChatExporter.Gui.Converters
[ValueConversion(typeof(bool), typeof(bool))]
public class InverseBoolConverter : IValueConverter
{
public static InverseBoolConverter Instance { get; } = new InverseBoolConverter();
public static InverseBoolConverter Instance { get; } = new();
public object Convert(object value, Type targetType, object parameter, CultureInfo culture)
{
@@ -7,7 +7,7 @@ namespace DiscordChatExporter.Gui.Converters
[ValueConversion(typeof(TimeSpan?), typeof(DateTime?))]
public class TimeSpanToDateTimeConverter : IValueConverter
{
public static TimeSpanToDateTimeConverter Instance { get; } = new TimeSpanToDateTimeConverter();
public static TimeSpanToDateTimeConverter Instance { get; } = new();
public object? Convert(object value, Type targetType, object parameter, CultureInfo culture)
{
@@ -1,5 +1,4 @@
<Project Sdk="Microsoft.NET.Sdk.WindowsDesktop">
<Import Project="../DiscordChatExporter.props" />
<PropertyGroup>
<OutputType>WinExe</OutputType>
@@ -17,12 +16,12 @@
<PackageReference Include="MaterialDesignColors" Version="1.2.2" />
<PackageReference Include="MaterialDesignThemes" Version="3.0.1" />
<PackageReference Include="Microsoft.Xaml.Behaviors.Wpf" Version="1.1.19" />
<PackageReference Include="Ookii.Dialogs.Wpf" Version="1.1.0" />
<PackageReference Include="Ookii.Dialogs.Wpf" Version="3.1.0" />
<PackageReference Include="Onova" Version="2.6.2" />
<PackageReference Include="Stylet" Version="1.3.4" />
<PackageReference Include="Stylet" Version="1.3.5" />
<PackageReference Include="Tyrrrz.Extensions" Version="1.6.5" />
<PackageReference Include="Tyrrrz.Settings" Version="1.3.4" />
<PackageReference Include="PropertyChanged.Fody" Version="3.2.10" PrivateAssets="all" />
<PackageReference Include="PropertyChanged.Fody" Version="3.3.1" PrivateAssets="all" />
</ItemGroup>
<ItemGroup>
@@ -1,8 +1,10 @@
using System;
using System.Collections.Generic;
using System.Linq;
using DiscordChatExporter.Domain.Discord;
using DiscordChatExporter.Domain.Discord.Models;
using DiscordChatExporter.Domain.Exporting;
using DiscordChatExporter.Domain.Utilities;
using DiscordChatExporter.Gui.Services;
using DiscordChatExporter.Gui.ViewModels.Framework;
@@ -82,8 +84,8 @@ namespace DiscordChatExporter.Gui.ViewModels.Dialogs
Guild!,
channel,
SelectedFormat,
After,
Before
After?.Pipe(Snowflake.FromDate),
Before?.Pipe(Snowflake.FromDate)
);
// Filter
@@ -211,8 +211,8 @@ namespace DiscordChatExporter.Gui.ViewModels
channel!,
dialog.OutputPath!,
dialog.SelectedFormat,
dialog.After,
dialog.Before,
dialog.After?.Pipe(Snowflake.FromDate),
dialog.Before?.Pipe(Snowflake.FromDate),
dialog.PartitionLimit,
dialog.ShouldDownloadMedia,
_settingsService.ShouldReuseMedia,
+1 -1
View File
@@ -8,7 +8,7 @@ Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Solution Items", "Solution
Changelog.md = Changelog.md
License.txt = License.txt
Readme.md = Readme.md
DiscordChatExporter.props = DiscordChatExporter.props
Directory.Build.props = Directory.Build.props
Dockerfile = Dockerfile
EndProjectSection
EndProject
+3 -3
View File
@@ -1,9 +1,9 @@
# Build
FROM mcr.microsoft.com/dotnet/sdk:3.1 AS build
FROM mcr.microsoft.com/dotnet/sdk:5.0 AS build
WORKDIR /src
COPY favicon.ico ./
COPY DiscordChatExporter.props ./
COPY Directory.Build.props ./
COPY DiscordChatExporter.Domain DiscordChatExporter.Domain
COPY DiscordChatExporter.Cli DiscordChatExporter.Cli
@@ -16,4 +16,4 @@ WORKDIR /app
COPY --from=build /src/DiscordChatExporter.Cli/publish ./
WORKDIR /app/out
ENTRYPOINT ["dotnet", "/app/DiscordChatExporter.Cli.dll"]
ENTRYPOINT ["dotnet", "/app/DiscordChatExporter.Cli.dll"]
+18 -11
View File
@@ -6,24 +6,31 @@
[![Donate](https://img.shields.io/badge/donate-$$$-purple.svg)](https://tyrrrz.me/donate)
[![Extra Services](https://img.shields.io/badge/extra%20services-xs:code-blue.svg)](https://xscode.com/Tyrrrz/DiscordChatExporter)
**Project status: active**.
**Project status: active**.
DiscordChatExporter can be used to export message history from a [Discord](https://discord.com) channel to a file. It works with direct messages, group messages, server channels, supports Discord's dialect of markdown and all other rich media features.
DiscordChatExporter can be used to export message history from a [Discord](https://discord.com) channel to a file. It works with direct messages, group messages, and server channels, and supports Discord's dialect of markdown as well as all other rich media features.
Have questions or need help? Check out the [wiki](https://github.com/Tyrrrz/DiscordChatExporter/wiki).
**If you have questions or issues, please check out the [wiki](https://github.com/Tyrrrz/DiscordChatExporter/wiki)**.
## Download
- **[Latest release](https://github.com/Tyrrrz/DiscordChatExporter/releases/latest)**
- [Docker image](https://hub.docker.com/r/tyrrrz/discordchatexporter): `docker pull tyrrrz/discordchatexporter`
- [AUR](https://aur.archlinux.org/packages/discord-chat-exporter-cli/): `yay -S discord-chat-exporter-cli`
- [CI build](https://github.com/Tyrrrz/DiscordChatExporter/actions)
DiscordChatExporter comes in two flavors: graphical user interface (**GUI**) and command line interface (**CLI**).
Note: This application requires .NET Core v3.1 runtime in order to run:
- **Graphical user interface** (Windows-only)
- Recommended: [Download `DiscordChatExporter.zip` from latest release](https://github.com/Tyrrrz/DiscordChatExporter/releases/latest) [requires .NET runtime]
- [Download `DiscordChatExporter.zip` from latest CI build](https://github.com/Tyrrrz/DiscordChatExporter/actions?query=workflow%3ACI) [requires .NET runtime]
- **Command line interface** (Windows, Linux, and macOS)
- Recommended: [Download `DiscordChatExporter.CLI.zip` from latest release](https://github.com/Tyrrrz/DiscordChatExporter/releases/latest) [requires .NET runtime]
- [Download `DiscordChatExporter.CLI.zip` from latest CI build](https://github.com/Tyrrrz/DiscordChatExporter/actions?query=workflow%3ACI) [requires .NET runtime]
- [Install from Docker Hub](https://hub.docker.com/r/tyrrrz/discordchatexporter): `docker pull tyrrrz/discordchatexporter`
- [Install from Arch Linux User Repository](https://aur.archlinux.org/packages/discord-chat-exporter-cli): `yay -S discord-chat-exporter-cli` [community-maintained]
- .NET Runtime for Windows: [x64](https://dotnet.microsoft.com/download/dotnet-core/thank-you/runtime-desktop-3.1.0-windows-x64-installer) | [x86](https://dotnet.microsoft.com/download/dotnet-core/thank-you/runtime-desktop-3.1.0-windows-x86-installer)
- .NET Runtime for macOS (for CLI version only): [x64](https://dotnet.microsoft.com/download/dotnet-core/thank-you/runtime-3.1.0-macos-x64-installer)
- .NET Runtime for Linux (for CLI version only): [Ubuntu/Debian](https://docs.microsoft.com/en-us/dotnet/core/install/linux-package-manager-ubuntu-1904#install-the-net-core-runtime)
**Important**: This application requires **.NET Core v3.1** runtime in order to run. Some download options come with the runtime pre-packaged, but those marked with `[requires .NET runtime]` do not. To install the runtime, find a suitable download option below:
- Windows: [x64](https://dotnet.microsoft.com/download/dotnet-core/thank-you/runtime-desktop-3.1.0-windows-x64-installer) | [x86](https://dotnet.microsoft.com/download/dotnet-core/thank-you/runtime-desktop-3.1.0-windows-x86-installer)
- macOS: [x64](https://dotnet.microsoft.com/download/dotnet-core/thank-you/runtime-3.1.0-macos-x64-installer)
- Linux: [find your distribution here](https://docs.microsoft.com/en-us/dotnet/core/install/linux)
## Features