Compare commits

..

34 Commits

Author SHA1 Message Date
Tyrrrz 6f8749a105 Update version 2023-11-09 13:11:59 +02:00
Tyrrrz 4896d748aa Update NuGet packages 2023-11-09 13:06:00 +02:00
Tyrrrz 4876ed192a Use ISO 8601 date format in CSV
Closes #1152
2023-11-07 22:16:25 +02:00
Tyrrrz 4f6e6f4f0b Write embed videos in JSON export
Closes #1120
2023-11-07 16:03:39 +02:00
Tyrrrz 8c75f04115 Add labels to the dockerfile 2023-11-07 15:09:31 +02:00
Tyrrrz df04b123cd Reorder props 2023-11-01 15:30:15 +02:00
Tyrrrz 578ed9be17 Prepare for .NET 8 2023-10-31 23:40:41 +02:00
Tyrrrz ebceacc707 Minor workflow cleanup 2023-10-29 23:59:09 +02:00
Tyrrrz eaab5786f2 Workflow improvements 2023-10-29 01:24:47 +03:00
Tyrrrz ebf7d65334 Fix step name in deploy job 2023-10-26 16:15:09 +03:00
Tyrrrz 52a23828ce Update version 2023-10-26 16:06:53 +03:00
Tyrrrz 2edd781e32 Update NuGet packages 2023-10-26 15:59:56 +03:00
Tyrrrz 19cdf47c6e Consolidate some docker jobs 2023-10-26 03:11:07 +03:00
Tyrrrz 65e0a63bda Consistency with other projects 2023-10-26 02:49:26 +03:00
Tyrrrz 692438d10c Reduce repetition in workflows using matrices 2023-10-26 02:45:20 +03:00
Tyrrrz c607c6f307 Fix deploy job 2023-10-26 00:03:55 +03:00
Tyrrrz aae43821e7 Simplify CI workflow 2023-10-24 15:13:30 +03:00
Tyrrrz 7b9a4e1aff Fix closing backticks in docs 2023-10-24 14:32:29 +03:00
LostXOR d8e57f94a8 Add command to fix broken permissions to Linux installation instructions (#1148)
Co-authored-by: Oleksii Holub <1935960+Tyrrrz@users.noreply.github.com>
2023-10-24 00:26:52 +03:00
Tyrrrz 7b7df564ed Allow empty names in interactions 2023-10-22 19:22:51 +03:00
Tyrrrz 80b59d1254 Update version 2023-10-19 18:52:19 +03:00
Tyrrrz 6579563816 Don't unfurl URLs in notify job 2023-10-17 16:18:44 +03:00
Tyrrrz e2971ed3a6 Improve notify job message 2023-10-17 15:01:59 +03:00
Tyrrrz a0a263f074 Improve error message for forum channels
Related to #1144
2023-10-16 17:22:28 +03:00
Tyrrrz 1438fd9f77 Temp fix for the docs link in the readme 2023-10-16 17:12:40 +03:00
Tyrrrz b298b45cf7 Escape markup in CLI 2023-10-13 01:17:13 +03:00
Tyrrrz 8d01c6be35 Improve notify message format 2023-10-12 16:00:47 +03:00
Tyrrrz ce4dbb2616 Update version 2023-10-12 15:43:31 +03:00
Tyrrrz 033d83bc4f More cleanup 2023-10-11 01:20:30 +03:00
Tyrrrz fb6cde38b6 Clean up after last commit 2023-10-10 00:04:57 +03:00
Adam Slatinský 46ede471a9 Remove CDN signature before hashing asset URL (#1138) 2023-10-09 23:58:31 +03:00
Tyrrrz 09f8937d99 Refactor 2023-10-09 16:18:56 +03:00
Adam Slatinský ad2dab2157 Add progress to fetching channels step (#1131) 2023-10-08 23:56:39 +03:00
Tyrrrz c5e426289f Include release link in notify 2023-10-05 22:52:31 +03:00
43 changed files with 631 additions and 326 deletions
+6
View File
@@ -15,6 +15,12 @@ Please follow the [instructions provided here](https://github.com/Tyrrrz/Discord
dotnet DiscordChatExporter.Cli.dll export -t TOKEN -c CHANNEL dotnet DiscordChatExporter.Cli.dll export -t TOKEN -c CHANNEL
``` ```
If the above command throws a "Permission denied" error, execute this command to fix the permissions:
```console
chmod 644 *.dll DiscordChatExporter.*
```
> [How to get Token and Channel IDs](https://github.com/Tyrrrz/DiscordChatExporter/blob/master/.docs/Token-and-IDs.md). > [How to get Token and Channel IDs](https://github.com/Tyrrrz/DiscordChatExporter/blob/master/.docs/Token-and-IDs.md).
There's much more you can do with DCE.CLI! Read the [CLI explained](https://github.com/Tyrrrz/DiscordChatExporter/blob/master/.docs/Getting-started.md#dcecli-commands-) page to get started. There's much more you can do with DCE.CLI! Read the [CLI explained](https://github.com/Tyrrrz/DiscordChatExporter/blob/master/.docs/Getting-started.md#dcecli-commands-) page to get started.
+11 -31
View File
@@ -1,8 +1,12 @@
name: docker name: docker
on: [push, pull_request] on:
push:
pull_request:
jobs: jobs:
# Outputs from this job aren't really used, but it's here to verify that
# the Dockerfile builds correctly on pull requests.
build: build:
runs-on: ubuntu-latest runs-on: ubuntu-latest
permissions: permissions:
@@ -29,9 +33,11 @@ jobs:
with: with:
name: DiscordChatExporter.Cli.Docker name: DiscordChatExporter.Cli.Docker
path: DiscordChatExporter.Cli.Docker.tar path: DiscordChatExporter.Cli.Docker.tar
if-no-files-found: error
deploy-latest: deploy:
if: ${{ github.event_name == 'push' && github.ref_type == 'branch' && github.ref_name == 'master' }} # Deploy to DockerHub only on tag push or master branch push
if: ${{ github.event_name == 'push' && (github.ref_type == 'tag' || github.ref_type == 'branch' && github.ref_name == 'master') }}
runs-on: ubuntu-latest runs-on: ubuntu-latest
permissions: permissions:
contents: read contents: read
@@ -55,32 +61,6 @@ jobs:
--platform linux/amd64,linux/arm64 --platform linux/amd64,linux/arm64
--push --push
--tag tyrrrz/discordchatexporter:latest --tag tyrrrz/discordchatexporter:latest
${{ github.ref_type == 'tag' && '--tag tyrrrz/discordchatexporter:$GITHUB_REF_NAME' || '' }}
${{ github.ref_type == 'tag' && '--tag tyrrrz/discordchatexporter:stable' || '' }}
. .
deploy-stable:
if: ${{ github.event_name == 'push' && github.ref_type == 'tag' }}
runs-on: ubuntu-latest
permissions:
contents: read
steps:
- name: Checkout
uses: actions/checkout@8ade135a41bc03ea155e62e844d188df1ea18608 # 4.1.0
- name: Install Docker Buildx
uses: docker/setup-buildx-action@f95db51fddba0c2d1ec667646a06c2ce06100226 # 3.0.0
- name: Login to DockerHub
run: >
echo ${{ secrets.DOCKER_TOKEN }} |
docker login --username tyrrrz --password-stdin
- name: Build & push image
run: >
docker buildx build
--file DiscordChatExporter.Cli.dockerfile
--platform linux/amd64,linux/arm64
--push
--tag tyrrrz/discordchatexporter:stable
--tag tyrrrz/discordchatexporter:${{ github.ref_name }}
.
+63 -55
View File
@@ -1,6 +1,8 @@
name: main name: main
on: [push, pull_request] on:
push:
pull_request:
env: env:
DOTNET_SKIP_FIRST_TIME_EXPERIENCE: true DOTNET_SKIP_FIRST_TIME_EXPERIENCE: true
@@ -25,6 +27,8 @@ jobs:
- name: Run tests - name: Run tests
# Tests need access to secrets, so we can't run them against PRs because of limited trust # Tests need access to secrets, so we can't run them against PRs because of limited trust
if: ${{ github.event_name != 'pull_request' }} if: ${{ github.event_name != 'pull_request' }}
env:
DISCORD_TOKEN: ${{ secrets.DISCORD_TOKEN }}
run: > run: >
dotnet test dotnet test
--configuration Release --configuration Release
@@ -33,8 +37,6 @@ jobs:
-- --
RunConfiguration.CollectSourceInformation=true RunConfiguration.CollectSourceInformation=true
DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=opencover DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=opencover
env:
DISCORD_TOKEN: ${{ secrets.DISCORD_TOKEN }}
- name: Upload coverage - name: Upload coverage
uses: codecov/codecov-action@eaaf4bedf32dbdc6b720b63067d99c4d77d6047d # 3.1.4 uses: codecov/codecov-action@eaaf4bedf32dbdc6b720b63067d99c4d77d6047d # 3.1.4
@@ -42,6 +44,12 @@ jobs:
token: ${{ secrets.CODECOV_TOKEN }} token: ${{ secrets.CODECOV_TOKEN }}
pack: pack:
strategy:
matrix:
app:
- DiscordChatExporter.Cli
- DiscordChatExporter.Gui
runs-on: windows-latest runs-on: windows-latest
permissions: permissions:
actions: write actions: write
@@ -56,80 +64,80 @@ jobs:
with: with:
dotnet-version: 7.0.x dotnet-version: 7.0.x
- name: Publish (CLI) - name: Publish app
run: > run: >
dotnet publish DiscordChatExporter.Cli dotnet publish ${{ matrix.app }}
--output DiscordChatExporter.Cli/publish/ --output ${{ matrix.app }}/bin/publish/
--configuration Release --configuration Release
- name: Publish (GUI) - name: Upload artifacts
run: >
dotnet publish DiscordChatExporter.Gui
--output DiscordChatExporter.Gui/publish/
--configuration Release
- name: Upload artifacts (CLI)
uses: actions/upload-artifact@a8a3f3ad30e3422c9c7b888a15615d19a852ae32 # 3.1.3 uses: actions/upload-artifact@a8a3f3ad30e3422c9c7b888a15615d19a852ae32 # 3.1.3
with: with:
name: DiscordChatExporter.Cli name: ${{ matrix.app }}
path: DiscordChatExporter.Cli/publish/ path: ${{ matrix.app }}/bin/publish/
if-no-files-found: error
- name: Upload artifacts (GUI) release:
uses: actions/upload-artifact@a8a3f3ad30e3422c9c7b888a15615d19a852ae32 # 3.1.3
with:
name: DiscordChatExporter
path: DiscordChatExporter.Gui/publish/
deploy:
if: ${{ github.event_name == 'push' && github.ref_type == 'tag' }} if: ${{ github.event_name == 'push' && github.ref_type == 'tag' }}
needs: needs:
- test - test
- pack - pack
runs-on: ubuntu-latest
permissions:
contents: write
steps:
- name: Create release
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: >
gh release create ${{ github.ref_name }}
--repo ${{ github.event.repository.full_name }}
--title ${{ github.ref_name }}
--notes "[Changelog](${{ github.event.repository.html_url }}/blob/${{ github.ref_name }}/Changelog.md)"
--verify-tag
deploy:
needs: release
strategy:
matrix:
app:
- DiscordChatExporter.Cli
- DiscordChatExporter.Gui
include:
- app: DiscordChatExporter.Cli
asset: DiscordChatExporter.Cli
- app: DiscordChatExporter.Gui
# GUI asset isn't suffixed, unlike the CLI asset
asset: DiscordChatExporter
runs-on: ubuntu-latest runs-on: ubuntu-latest
permissions: permissions:
actions: read actions: read
contents: write contents: write
steps: steps:
- name: Download artifacts (CLI) - name: Download artifacts
uses: actions/download-artifact@9bc31d5ccc31df68ecc42ccf4149144866c47d8a # 3.0.2 uses: actions/download-artifact@9bc31d5ccc31df68ecc42ccf4149144866c47d8a # 3.0.2
with: with:
name: DiscordChatExporter.Cli name: ${{ matrix.app }}
path: DiscordChatExporter.Cli path: ${{ matrix.app }}/
- name: Download artifacts (GUI) - name: Create package
uses: actions/download-artifact@9bc31d5ccc31df68ecc42ccf4149144866c47d8a # 3.0.2 # Change into the artifacts directory to avoid including the directory itself in the zip archive
with: working-directory: ${{ matrix.app }}/
name: DiscordChatExporter run: zip -r ../${{ matrix.asset }}.zip .
path: DiscordChatExporter.Gui
- name: Create package (CLI) - name: Upload release asset
shell: pwsh
run: >
Compress-Archive
-Path DiscordChatExporter.Cli/*
-DestinationPath DiscordChatExporter.Cli.zip
-Force
- name: Create package (GUI)
shell: pwsh
run: >
Compress-Archive
-Path DiscordChatExporter.Gui/*
-DestinationPath DiscordChatExporter.zip
-Force
- name: Create release
env: env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: > run: >
gh release create "${{ github.ref_name }}" gh release upload ${{ github.ref_name }}
"DiscordChatExporter.Cli.zip" ${{ matrix.asset }}.zip
"DiscordChatExporter.zip" --repo ${{ github.event.repository.full_name }}
--repo "${{ github.event.repository.full_name }}"
--title "${{ github.ref_name }}"
--notes "[Changelog](${{ github.event.repository.html_url }}/blob/${{ github.ref_name }}/Changelog.md)"
--verify-tag
notify: notify:
needs: deploy needs: deploy
@@ -148,6 +156,6 @@ jobs:
body: | body: |
{ {
"avatar_url": "https://raw.githubusercontent.com/${{ github.event.repository.full_name }}/${{ github.ref_name }}/favicon.png", "avatar_url": "https://raw.githubusercontent.com/${{ github.event.repository.full_name }}/${{ github.ref_name }}/favicon.png",
"content": "**${{ github.event.repository.name }}** new version released!\nVersion: `${{ github.ref_name }}`\nChangelog: <${{ github.event.repository.html_url }}/blob/${{ github.ref_name }}/Changelog.md>" "content": "**${{ github.event.repository.name }}** v${{ github.ref_name }} has been released!\n🔗 [Download](<${{ github.event.repository.html_url }}/releases/tag/${{ github.ref_name }}>) • [Changelog](<${{ github.event.repository.html_url }}/blob/${{ github.ref_name }}/Changelog.md>)"
} }
retry-count: 5 retry-count: 5
+26
View File
@@ -1,5 +1,31 @@
# Changelog # Changelog
## v2.42.3 (09-Nov-2023)
- JSON changes:
- Added the `video` field to the embed object, which may contain additional information about an embedded video resource.
- CSV changes:
- Fixed an issue where the `Date` field was incorrectly formatted. Now, the dates follow the `ISO 8601` standard.
## v2.42.2 (26-Oct-2023)
- General changes:
- Fixed an issue where trying to export a channel that contained application interactions with empty command names crashed the application.
## v2.42.1 (19-Oct-2023)
- General changes:
- Improved the error message shown when trying to export a forum channel. Such channels cannot be exported directly, you need to fetch and export their individual threads instead.
- CLI changes:
- Fixed an issue where fetching threads with `[` or `]` in their name crashed the application.
## v2.42 (12-Oct-2023)
- General changes:
- Fixed an issue where the "reuse assets" option did not work as intended due to Discord recently introducing signatures in the CDN URLs. (Thanks [@slatinsky](https://github.com/slatinsky))
- CLI changes:
- Added progress reporting for the "fetching channels" stage of the `exportguild` and `exportall` commands. (Thanks [@slatinsky](https://github.com/slatinsky))
## v2.41.2 (05-Oct-2023) ## v2.41.2 (05-Oct-2023)
- General changes: - General changes:
+1 -1
View File
@@ -2,7 +2,7 @@
<PropertyGroup> <PropertyGroup>
<TargetFramework>net7.0</TargetFramework> <TargetFramework>net7.0</TargetFramework>
<Version>2.41.2</Version> <Version>2.42.3</Version>
<Company>Tyrrrz</Company> <Company>Tyrrrz</Company>
<Copyright>Copyright (c) Oleksii Holub</Copyright> <Copyright>Copyright (c) Oleksii Holub</Copyright>
<LangVersion>preview</LangVersion> <LangVersion>preview</LangVersion>
@@ -11,19 +11,19 @@
</ItemGroup> </ItemGroup>
<ItemGroup> <ItemGroup>
<PackageReference Include="AngleSharp" Version="1.0.4" /> <PackageReference Include="AngleSharp" Version="1.0.5" />
<PackageReference Include="coverlet.collector" Version="6.0.0" PrivateAssets="all" /> <PackageReference Include="coverlet.collector" Version="6.0.0" PrivateAssets="all" />
<PackageReference Include="CSharpier.MsBuild" Version="0.25.0" PrivateAssets="all" /> <PackageReference Include="CSharpier.MsBuild" Version="0.26.1" PrivateAssets="all" />
<PackageReference Include="FluentAssertions" Version="6.12.0" /> <PackageReference Include="FluentAssertions" Version="6.12.0" />
<PackageReference Include="GitHubActionsTestLogger" Version="2.3.3" PrivateAssets="all" /> <PackageReference Include="GitHubActionsTestLogger" Version="2.3.3" PrivateAssets="all" />
<PackageReference Include="JsonExtensions" Version="1.2.0" /> <PackageReference Include="JsonExtensions" Version="1.2.0" />
<PackageReference Include="Microsoft.Extensions.Configuration" Version="7.0.0" /> <PackageReference Include="Microsoft.Extensions.Configuration" Version="7.0.0" />
<PackageReference Include="Microsoft.Extensions.Configuration.EnvironmentVariables" Version="7.0.0" /> <PackageReference Include="Microsoft.Extensions.Configuration.EnvironmentVariables" Version="7.0.0" />
<PackageReference Include="Microsoft.Extensions.Configuration.UserSecrets" Version="7.0.0" /> <PackageReference Include="Microsoft.Extensions.Configuration.UserSecrets" Version="7.0.0" />
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.7.2" /> <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.8.0" />
<PackageReference Include="ReflectionMagic" Version="5.0.0" /> <PackageReference Include="ReflectionMagic" Version="5.0.0" />
<PackageReference Include="xunit" Version="2.5.1" /> <PackageReference Include="xunit" Version="2.6.1" />
<PackageReference Include="xunit.runner.visualstudio" Version="2.5.1" PrivateAssets="all" /> <PackageReference Include="xunit.runner.visualstudio" Version="2.5.3" PrivateAssets="all" />
</ItemGroup> </ItemGroup>
<ItemGroup> <ItemGroup>
+11 -4
View File
@@ -12,7 +12,7 @@ FROM --platform=$BUILDPLATFORM mcr.microsoft.com/dotnet/sdk:8.0-preview AS build
# we can build the assembly for the correct platform. # we can build the assembly for the correct platform.
ARG TARGETARCH ARG TARGETARCH
WORKDIR /build WORKDIR /tmp/dce
COPY favicon.ico . COPY favicon.ico .
COPY NuGet.config . COPY NuGet.config .
@@ -22,17 +22,24 @@ COPY DiscordChatExporter.Cli DiscordChatExporter.Cli
# Publish a self-contained assembly so we can use a slimmer runtime image # Publish a self-contained assembly so we can use a slimmer runtime image
RUN dotnet publish DiscordChatExporter.Cli \ RUN dotnet publish DiscordChatExporter.Cli \
--configuration Release \
-p:CSharpier_Bypass=true \ -p:CSharpier_Bypass=true \
--configuration Release \
--self-contained \ --self-contained \
--use-current-runtime \ --use-current-runtime \
--arch $TARGETARCH \ --arch $TARGETARCH \
--output publish/ --output DiscordChatExporter.Cli/bin/publish/
# -- Run # -- Run
# Use `runtime-deps` instead of `runtime` because we have a self-contained assembly # Use `runtime-deps` instead of `runtime` because we have a self-contained assembly
FROM --platform=$TARGETPLATFORM mcr.microsoft.com/dotnet/runtime-deps:7.0 AS run FROM --platform=$TARGETPLATFORM mcr.microsoft.com/dotnet/runtime-deps:7.0 AS run
LABEL org.opencontainers.image.title="DiscordChatExporter.Cli"
LABEL org.opencontainers.image.description="DiscordChatExporter is an application that can be used to export message history from any Discord channel to a file."
LABEL org.opencontainers.image.authors="tyrrrz.me"
LABEL org.opencontainers.image.url="https://github.com/Tyrrrz/DiscordChatExporter"
LABEL org.opencontainers.image.source="https://github.com/Tyrrrz/DiscordChatExporter/blob/master/DiscordChatExporter.Cli.dockerfile"
LABEL org.opencontainers.image.licenses="MIT"
# Create a non-root user to run the app, so that the output files can be accessed by the host # Create a non-root user to run the app, so that the output files can be accessed by the host
# https://github.com/Tyrrrz/DiscordChatExporter/issues/851 # https://github.com/Tyrrrz/DiscordChatExporter/issues/851
RUN adduser --disabled-password --no-create-home dce RUN adduser --disabled-password --no-create-home dce
@@ -42,5 +49,5 @@ USER dce
# stays the same for backwards compatibility. # stays the same for backwards compatibility.
WORKDIR /out WORKDIR /out
COPY --from=build /build/publish /opt/dce COPY --from=build /tmp/dce/DiscordChatExporter.Cli/bin/publish /opt/dce
ENTRYPOINT ["/opt/dce/DiscordChatExporter.Cli"] ENTRYPOINT ["/opt/dce/DiscordChatExporter.Cli"]
@@ -37,11 +37,13 @@ public abstract class DiscordCommandBase : ICommand
{ {
using (console.WithForegroundColor(ConsoleColor.DarkYellow)) using (console.WithForegroundColor(ConsoleColor.DarkYellow))
{ {
console.Error.WriteLine( console
"Warning: Option --bot is deprecated and should not be used. " .Error
+ "The type of the provided token is now inferred automatically. " .WriteLine(
+ "Please update your workflows as this option may be completely removed in a future version." "Warning: Option --bot is deprecated and should not be used. "
); + "The type of the provided token is now inferred automatically. "
+ "Please update your workflows as this option may be completely removed in a future version."
);
} }
} }
#pragma warning restore CS0618 #pragma warning restore CS0618
@@ -186,7 +186,7 @@ public abstract class ExportCommandBase : DiscordCommandBase
// https://github.com/Tyrrrz/DiscordChatExporter/issues/1124 // https://github.com/Tyrrrz/DiscordChatExporter/issues/1124
ParallelLimit > 1 ParallelLimit > 1
) )
.StartAsync(async progressContext => .StartAsync(async ctx =>
{ {
await Parallel.ForEachAsync( await Parallel.ForEachAsync(
channels, channels,
@@ -199,8 +199,8 @@ public abstract class ExportCommandBase : DiscordCommandBase
{ {
try try
{ {
await progressContext.StartTaskAsync( await ctx.StartTaskAsync(
channel.GetHierarchicalName(), Markup.Escape(channel.GetHierarchicalName()),
async progress => async progress =>
{ {
var guild = await Discord.GetGuildAsync( var guild = await Discord.GetGuildAsync(
@@ -244,9 +244,11 @@ public abstract class ExportCommandBase : DiscordCommandBase
// Print the result // Print the result
using (console.WithForegroundColor(ConsoleColor.White)) using (console.WithForegroundColor(ConsoleColor.White))
{ {
await console.Output.WriteLineAsync( await console
$"Successfully exported {channels.Count - errorsByChannel.Count} channel(s)." .Output
); .WriteLineAsync(
$"Successfully exported {channels.Count - errorsByChannel.Count} channel(s)."
);
} }
// Print errors // Print errors
@@ -256,9 +258,11 @@ public abstract class ExportCommandBase : DiscordCommandBase
using (console.WithForegroundColor(ConsoleColor.Red)) using (console.WithForegroundColor(ConsoleColor.Red))
{ {
await console.Error.WriteLineAsync( await console
$"Failed to export {errorsByChannel.Count} channel(s):" .Error
); .WriteLineAsync(
$"Failed to export {errorsByChannel.Count} the following channel(s):"
);
} }
foreach (var (channel, error) in errorsByChannel) foreach (var (channel, error) in errorsByChannel)
@@ -320,33 +324,51 @@ public abstract class ExportCommandBase : DiscordCommandBase
// Support Ukraine callout // Support Ukraine callout
if (!IsUkraineSupportMessageDisabled) if (!IsUkraineSupportMessageDisabled)
{ {
console.Output.WriteLine( console
"┌────────────────────────────────────────────────────────────────────┐" .Output
); .WriteLine(
console.Output.WriteLine( "┌────────────────────────────────────────────────────────────────────┐"
"│ Thank you for supporting Ukraine <3 │" );
); console
console.Output.WriteLine( .Output
"│ │" .WriteLine(
); "│ Thank you for supporting Ukraine <3 │"
console.Output.WriteLine( );
"│ As Russia wages a genocidal war against my country, │" console
); .Output
console.Output.WriteLine( .WriteLine(
"│ I'm grateful to everyone who continues to │" "│ │"
); );
console.Output.WriteLine( console
"│ stand with Ukraine in our fight for freedom. │" .Output
); .WriteLine(
console.Output.WriteLine( "│ As Russia wages a genocidal war against my country, │"
"│ │" );
); console
console.Output.WriteLine( .Output
"│ Learn more: https://tyrrrz.me/ukraine │" .WriteLine(
); "│ I'm grateful to everyone who continues to │"
console.Output.WriteLine( );
"└────────────────────────────────────────────────────────────────────┘" console
); .Output
.WriteLine(
"│ stand with Ukraine in our fight for freedom. │"
);
console
.Output
.WriteLine(
"│ │"
);
console
.Output
.WriteLine(
"│ Learn more: https://tyrrrz.me/ukraine │"
);
console
.Output
.WriteLine(
"└────────────────────────────────────────────────────────────────────┘"
);
console.Output.WriteLine(""); console.Output.WriteLine("");
} }
@@ -1,17 +1,17 @@
using System;
using System.Collections.Generic; using System.Collections.Generic;
using System.IO.Compression; using System.Linq;
using System.Text.Json;
using System.Threading.Tasks; using System.Threading.Tasks;
using CliFx.Attributes; using CliFx.Attributes;
using CliFx.Exceptions;
using CliFx.Infrastructure; using CliFx.Infrastructure;
using DiscordChatExporter.Cli.Commands.Base; using DiscordChatExporter.Cli.Commands.Base;
using DiscordChatExporter.Cli.Commands.Converters; using DiscordChatExporter.Cli.Commands.Converters;
using DiscordChatExporter.Cli.Commands.Shared; using DiscordChatExporter.Cli.Commands.Shared;
using DiscordChatExporter.Core.Discord; using DiscordChatExporter.Cli.Utils.Extensions;
using DiscordChatExporter.Core.Discord.Data; using DiscordChatExporter.Core.Discord.Data;
using DiscordChatExporter.Core.Discord.Dump;
using DiscordChatExporter.Core.Exceptions; using DiscordChatExporter.Core.Exceptions;
using JsonExtensions.Reading; using Spectre.Console;
namespace DiscordChatExporter.Cli.Commands; namespace DiscordChatExporter.Cli.Commands;
@@ -51,39 +51,84 @@ public class ExportAllCommand : ExportCommandBase
// Pull from the API // Pull from the API
if (string.IsNullOrWhiteSpace(DataPackageFilePath)) if (string.IsNullOrWhiteSpace(DataPackageFilePath))
{ {
await console.Output.WriteLineAsync("Fetching channels...");
await foreach (var guild in Discord.GetUserGuildsAsync(cancellationToken)) await foreach (var guild in Discord.GetUserGuildsAsync(cancellationToken))
{ {
// Regular channels // Regular channels
await foreach ( await console
var channel in Discord.GetGuildChannelsAsync(guild.Id, cancellationToken) .Output
) .WriteLineAsync($"Fetching channels for guild '{guild.Name}'...");
{
if (channel.IsCategory)
continue;
if (!IncludeVoiceChannels && channel.IsVoice) var fetchedChannelsCount = 0;
continue; await console
.CreateStatusTicker()
.StartAsync(
"...",
async ctx =>
{
await foreach (
var channel in Discord.GetGuildChannelsAsync(
guild.Id,
cancellationToken
)
)
{
if (channel.IsCategory)
continue;
channels.Add(channel); if (!IncludeVoiceChannels && channel.IsVoice)
} continue;
channels.Add(channel);
ctx.Status(
Markup.Escape($"Fetched '{channel.GetHierarchicalName()}'.")
);
fetchedChannelsCount++;
}
}
);
await console.Output.WriteLineAsync($"Fetched {fetchedChannelsCount} channel(s).");
// Threads // Threads
if (ThreadInclusionMode != ThreadInclusionMode.None) if (ThreadInclusionMode != ThreadInclusionMode.None)
{ {
await foreach ( await console
var thread in Discord.GetGuildThreadsAsync( .Output
guild.Id, .WriteLineAsync($"Fetching threads for guild '{guild.Name}'...");
ThreadInclusionMode == ThreadInclusionMode.All,
Before, var fetchedThreadsCount = 0;
After, await console
cancellationToken .CreateStatusTicker()
) .StartAsync(
) "...",
{ async ctx =>
channels.Add(thread); {
} await foreach (
var thread in Discord.GetGuildThreadsAsync(
guild.Id,
ThreadInclusionMode == ThreadInclusionMode.All,
Before,
After,
cancellationToken
)
)
{
channels.Add(thread);
ctx.Status(
Markup.Escape($"Fetched '{thread.GetHierarchicalName()}'.")
);
fetchedThreadsCount++;
}
}
);
await console
.Output
.WriteLineAsync($"Fetched {fetchedThreadsCount} thread(s).");
} }
} }
} }
@@ -91,39 +136,59 @@ public class ExportAllCommand : ExportCommandBase
else else
{ {
await console.Output.WriteLineAsync("Extracting channels..."); await console.Output.WriteLineAsync("Extracting channels...");
using var archive = ZipFile.OpenRead(DataPackageFilePath);
var entry = archive.GetEntry("messages/index.json"); var dump = await DataDump.LoadAsync(DataPackageFilePath, cancellationToken);
if (entry is null) var inaccessibleChannels = new List<DataDumpChannel>();
throw new CommandException("Could not find channel index inside the data package.");
await using var stream = entry.Open(); await console
using var document = await JsonDocument.ParseAsync(stream, default, cancellationToken); .CreateStatusTicker()
.StartAsync(
"...",
async ctx =>
{
foreach (var dumpChannel in dump.Channels)
{
ctx.Status(
Markup.Escape(
$"Fetching '{dumpChannel.Name}' ({dumpChannel.Id})..."
)
);
foreach (var property in document.RootElement.EnumerateObjectOrEmpty()) try
{ {
var channelId = Snowflake.Parse(property.Name); var channel = await Discord.GetChannelAsync(
var channelName = property.Value.GetString(); dumpChannel.Id,
cancellationToken
);
// Null items refer to deleted channels channels.Add(channel);
if (channelName is null) }
continue; catch (DiscordChatExporterException)
{
await console.Output.WriteLineAsync( inaccessibleChannels.Add(dumpChannel);
$"Fetching channel '{channelName}' ({channelId})..." }
}
}
); );
try await console.Output.WriteLineAsync($"Fetched {channels} channel(s).");
// Print inaccessible channels
if (inaccessibleChannels.Any())
{
await console.Output.WriteLineAsync();
using (console.WithForegroundColor(ConsoleColor.Red))
{ {
var channel = await Discord.GetChannelAsync(channelId, cancellationToken); await console
channels.Add(channel); .Error
} .WriteLineAsync("Failed to access the following channel(s):");
catch (DiscordChatExporterException)
{
await console.Error.WriteLineAsync(
$"Channel '{channelName}' ({channelId}) is inaccessible."
);
} }
foreach (var dumpChannel in inaccessibleChannels)
await console.Error.WriteLineAsync($"{dumpChannel.Name} ({dumpChannel.Id})");
await console.Error.WriteLineAsync();
} }
} }
@@ -1,12 +1,15 @@
using System.Collections.Generic; using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks; using System.Threading.Tasks;
using CliFx.Attributes; using CliFx.Attributes;
using CliFx.Infrastructure; using CliFx.Infrastructure;
using DiscordChatExporter.Cli.Commands.Base; using DiscordChatExporter.Cli.Commands.Base;
using DiscordChatExporter.Cli.Commands.Converters; using DiscordChatExporter.Cli.Commands.Converters;
using DiscordChatExporter.Cli.Commands.Shared; using DiscordChatExporter.Cli.Commands.Shared;
using DiscordChatExporter.Cli.Utils.Extensions;
using DiscordChatExporter.Core.Discord; using DiscordChatExporter.Core.Discord;
using DiscordChatExporter.Core.Discord.Data; using DiscordChatExporter.Core.Discord.Data;
using Spectre.Console;
namespace DiscordChatExporter.Cli.Commands; namespace DiscordChatExporter.Cli.Commands;
@@ -33,35 +36,69 @@ public class ExportGuildCommand : ExportCommandBase
var cancellationToken = console.RegisterCancellationHandler(); var cancellationToken = console.RegisterCancellationHandler();
var channels = new List<Channel>(); var channels = new List<Channel>();
// Regular channels
await console.Output.WriteLineAsync("Fetching channels..."); await console.Output.WriteLineAsync("Fetching channels...");
// Regular channels var fetchedChannelsCount = 0;
await foreach (var channel in Discord.GetGuildChannelsAsync(GuildId, cancellationToken)) await console
{ .CreateStatusTicker()
if (channel.IsCategory) .StartAsync(
continue; "...",
async ctx =>
{
await foreach (
var channel in Discord.GetGuildChannelsAsync(GuildId, cancellationToken)
)
{
if (channel.IsCategory)
continue;
if (!IncludeVoiceChannels && channel.IsVoice) if (!IncludeVoiceChannels && channel.IsVoice)
continue; continue;
channels.Add(channel); channels.Add(channel);
}
ctx.Status(Markup.Escape($"Fetched '{channel.GetHierarchicalName()}'."));
fetchedChannelsCount++;
}
}
);
await console.Output.WriteLineAsync($"Fetched {fetchedChannelsCount} channel(s).");
// Threads // Threads
if (ThreadInclusionMode != ThreadInclusionMode.None) if (ThreadInclusionMode != ThreadInclusionMode.None)
{ {
await foreach ( await console.Output.WriteLineAsync("Fetching threads...");
var thread in Discord.GetGuildThreadsAsync(
GuildId, var fetchedThreadsCount = 0;
ThreadInclusionMode == ThreadInclusionMode.All, await console
Before, .CreateStatusTicker()
After, .StartAsync(
cancellationToken "...",
) async ctx =>
) {
{ await foreach (
channels.Add(thread); var thread in Discord.GetGuildThreadsAsync(
} GuildId,
ThreadInclusionMode == ThreadInclusionMode.All,
Before,
After,
cancellationToken
)
)
{
channels.Add(thread);
ctx.Status(Markup.Escape($"Fetched '{thread.GetHierarchicalName()}'."));
fetchedThreadsCount++;
}
}
);
await console.Output.WriteLineAsync($"Fetched {fetchedThreadsCount} thread(s).");
} }
await ExportAsync(console, channels); await ExportAsync(console, channels);
@@ -64,9 +64,9 @@ public class GetChannelsCommand : DiscordCommandBase
foreach (var channel in channels) foreach (var channel in channels)
{ {
// Channel ID // Channel ID
await console.Output.WriteAsync( await console
channel.Id.ToString().PadRight(channelIdMaxLength, ' ') .Output
); .WriteAsync(channel.Id.ToString().PadRight(channelIdMaxLength, ' '));
// Separator // Separator
using (console.WithForegroundColor(ConsoleColor.DarkGray)) using (console.WithForegroundColor(ConsoleColor.DarkGray))
@@ -88,9 +88,11 @@ public class GetChannelsCommand : DiscordCommandBase
await console.Output.WriteAsync(" * "); await console.Output.WriteAsync(" * ");
// Thread ID // Thread ID
await console.Output.WriteAsync( await console
channelThread.Id.ToString().PadRight(channelThreadIdMaxLength, ' ') .Output
); .WriteAsync(
channelThread.Id.ToString().PadRight(channelThreadIdMaxLength, ' ')
);
// Separator // Separator
using (console.WithForegroundColor(ConsoleColor.DarkGray)) using (console.WithForegroundColor(ConsoleColor.DarkGray))
@@ -106,9 +108,9 @@ public class GetChannelsCommand : DiscordCommandBase
// Thread status // Thread status
using (console.WithForegroundColor(ConsoleColor.White)) using (console.WithForegroundColor(ConsoleColor.White))
await console.Output.WriteLineAsync( await console
channelThread.IsArchived ? "Archived" : "Active" .Output
); .WriteLineAsync(channelThread.IsArchived ? "Archived" : "Active");
} }
} }
} }
@@ -33,9 +33,9 @@ public class GetDirectChannelsCommand : DiscordCommandBase
foreach (var channel in channels) foreach (var channel in channels)
{ {
// Channel ID // Channel ID
await console.Output.WriteAsync( await console
channel.Id.ToString().PadRight(channelIdMaxLength, ' ') .Output
); .WriteAsync(channel.Id.ToString().PadRight(channelIdMaxLength, ' '));
// Separator // Separator
using (console.WithForegroundColor(ConsoleColor.DarkGray)) using (console.WithForegroundColor(ConsoleColor.DarkGray))
@@ -15,18 +15,20 @@ public class GuideCommand : ICommand
using (console.WithForegroundColor(ConsoleColor.White)) using (console.WithForegroundColor(ConsoleColor.White))
console.Output.WriteLine("To get user token:"); console.Output.WriteLine("To get user token:");
console.Output.WriteLine( console
" * Automating user accounts is technically against TOS — USE AT YOUR OWN RISK!" .Output
); .WriteLine(
" * Automating user accounts is technically against TOS — USE AT YOUR OWN RISK!"
);
console.Output.WriteLine(" 1. Open Discord in your web browser and login"); console.Output.WriteLine(" 1. Open Discord in your web browser and login");
console.Output.WriteLine(" 2. Open any server or direct message channel"); console.Output.WriteLine(" 2. Open any server or direct message channel");
console.Output.WriteLine(" 3. Press Ctrl+Shift+I to show developer tools"); console.Output.WriteLine(" 3. Press Ctrl+Shift+I to show developer tools");
console.Output.WriteLine(" 4. Navigate to the Network tab"); console.Output.WriteLine(" 4. Navigate to the Network tab");
console.Output.WriteLine(" 5. Press Ctrl+R to reload"); console.Output.WriteLine(" 5. Press Ctrl+R to reload");
console.Output.WriteLine(" 6. Switch between random channels to trigger network requests"); console.Output.WriteLine(" 6. Switch between random channels to trigger network requests");
console.Output.WriteLine( console
" 7. Search for a request containing \"messages?limit=50\" or similar" .Output
); .WriteLine(" 7. Search for a request containing \"messages?limit=50\" or similar");
console.Output.WriteLine(" 8. Select the Headers tab on the right"); console.Output.WriteLine(" 8. Select the Headers tab on the right");
console.Output.WriteLine(" 9. Scroll down to the Request Headers section"); console.Output.WriteLine(" 9. Scroll down to the Request Headers section");
console.Output.WriteLine(" 10. Copy the value of the \"authorization\" header"); console.Output.WriteLine(" 10. Copy the value of the \"authorization\" header");
@@ -40,9 +42,11 @@ public class GuideCommand : ICommand
console.Output.WriteLine(" 2. Open your application's settings"); console.Output.WriteLine(" 2. Open your application's settings");
console.Output.WriteLine(" 3. Navigate to the Bot section on the left"); console.Output.WriteLine(" 3. Navigate to the Bot section on the left");
console.Output.WriteLine(" 4. Under Token click Copy"); console.Output.WriteLine(" 4. Under Token click Copy");
console.Output.WriteLine( console
" * Your bot needs to have Message Content Intent enabled to read messages" .Output
); .WriteLine(
" * Your bot needs to have Message Content Intent enabled to read messages"
);
console.Output.WriteLine(); console.Output.WriteLine();
// Guild or channel ID // Guild or channel ID
@@ -53,20 +57,22 @@ public class GuideCommand : ICommand
console.Output.WriteLine(" 2. Open Settings"); console.Output.WriteLine(" 2. Open Settings");
console.Output.WriteLine(" 3. Go to Advanced section"); console.Output.WriteLine(" 3. Go to Advanced section");
console.Output.WriteLine(" 4. Enable Developer Mode"); console.Output.WriteLine(" 4. Enable Developer Mode");
console.Output.WriteLine( console
" 5. Right-click on the desired guild or channel and click Copy Server ID or Copy Channel ID" .Output
); .WriteLine(
" 5. Right-click on the desired guild or channel and click Copy Server ID or Copy Channel ID"
);
console.Output.WriteLine(); console.Output.WriteLine();
// Docs link // Docs link
using (console.WithForegroundColor(ConsoleColor.White)) using (console.WithForegroundColor(ConsoleColor.White))
console.Output.WriteLine( console
"If you have questions or issues, please refer to the documentation:" .Output
); .WriteLine("If you have questions or issues, please refer to the documentation:");
using (console.WithForegroundColor(ConsoleColor.DarkCyan)) using (console.WithForegroundColor(ConsoleColor.DarkCyan))
console.Output.WriteLine( console
"https://github.com/Tyrrrz/DiscordChatExporter/blob/master/.docs" .Output
); .WriteLine("https://github.com/Tyrrrz/DiscordChatExporter/blob/master/.docs");
return default; return default;
} }
@@ -7,7 +7,7 @@
<ItemGroup> <ItemGroup>
<PackageReference Include="CliFx" Version="2.3.4" /> <PackageReference Include="CliFx" Version="2.3.4" />
<PackageReference Include="CSharpier.MsBuild" Version="0.25.0" PrivateAssets="all" /> <PackageReference Include="CSharpier.MsBuild" Version="0.26.1" PrivateAssets="all" />
<PackageReference Include="Deorcify" Version="1.0.2" PrivateAssets="all" /> <PackageReference Include="Deorcify" Version="1.0.2" PrivateAssets="all" />
<PackageReference Include="DotnetRuntimeBootstrapper" Version="2.5.1" PrivateAssets="all" /> <PackageReference Include="DotnetRuntimeBootstrapper" Version="2.5.1" PrivateAssets="all" />
<PackageReference Include="Gress" Version="2.1.1" /> <PackageReference Include="Gress" Version="2.1.1" />
@@ -17,6 +17,9 @@ internal static class ConsoleExtensions
} }
); );
public static Status CreateStatusTicker(this IConsole console) =>
console.CreateAnsiConsole().Status().AutoRefresh(true);
public static Progress CreateProgressTicker(this IConsole console) => public static Progress CreateProgressTicker(this IConsole console) =>
console console
.CreateAnsiConsole() .CreateAnsiConsole()
@@ -31,18 +34,17 @@ internal static class ConsoleExtensions
); );
public static async ValueTask StartTaskAsync( public static async ValueTask StartTaskAsync(
this ProgressContext progressContext, this ProgressContext context,
string description, string description,
Func<ProgressTask, ValueTask> performOperationAsync Func<ProgressTask, ValueTask> performOperationAsync
) )
{ {
// Description cannot be empty // Description cannot be empty
// https://github.com/Tyrrrz/DiscordChatExporter/issues/1133 // https://github.com/Tyrrrz/DiscordChatExporter/issues/1133
var actualDescription = !string.IsNullOrWhiteSpace(description) ? description : "?"; var actualDescription = !string.IsNullOrWhiteSpace(description) ? description : "...";
var progressTask = progressContext.AddTask( var progressTask = context.AddTask(
// Don't recognize random square brackets as style tags actualDescription,
Markup.Escape(actualDescription),
new ProgressTaskSettings { MaxValue = 1 } new ProgressTaskSettings { MaxValue = 1 }
); );
@@ -1,5 +1,5 @@
using JsonExtensions.Reading;
using System.Text.Json; using System.Text.Json;
using JsonExtensions.Reading;
namespace DiscordChatExporter.Core.Discord.Data.Embeds; namespace DiscordChatExporter.Core.Discord.Data.Embeds;
@@ -12,9 +12,11 @@ public partial record SpotifyTrackEmbedProjection
private static string? TryParseTrackId(string embedUrl) private static string? TryParseTrackId(string embedUrl)
{ {
// https://open.spotify.com/track/1LHZMWefF9502NPfArRfvP?si=3efac6ce9be04f0a // https://open.spotify.com/track/1LHZMWefF9502NPfArRfvP?si=3efac6ce9be04f0a
var trackId = Regex.Match(embedUrl, @"spotify\.com/track/(.*?)(?:\?|&|/|$)").Groups[ var trackId = Regex
1 .Match(embedUrl, @"spotify\.com/track/(.*?)(?:\?|&|/|$)")
].Value; .Groups[1]
.Value;
if (!string.IsNullOrWhiteSpace(trackId)) if (!string.IsNullOrWhiteSpace(trackId))
return trackId; return trackId;
@@ -10,7 +10,7 @@ public record Interaction(Snowflake Id, string Name, User User)
public static Interaction Parse(JsonElement json) public static Interaction Parse(JsonElement json)
{ {
var id = json.GetProperty("id").GetNonWhiteSpaceString().Pipe(Snowflake.Parse); var id = json.GetProperty("id").GetNonWhiteSpaceString().Pipe(Snowflake.Parse);
var name = json.GetProperty("name").GetNonWhiteSpaceString(); var name = json.GetProperty("name").GetNonNullString(); // may be empty, but not null
var user = json.GetProperty("user").Pipe(User.Parse); var user = json.GetProperty("user").Pipe(User.Parse);
return new Interaction(id, name, user); return new Interaction(id, name, user);
@@ -99,7 +99,8 @@ public partial record Message
if (trailingEmbeds.Any()) if (trailingEmbeds.Any())
{ {
// Concatenate all images into one embed // Concatenate all images into one embed
var images = embed.Images var images = embed
.Images
.Concat(trailingEmbeds.SelectMany(e => e.Images)) .Concat(trailingEmbeds.SelectMany(e => e.Images))
.ToArray(); .ToArray();
@@ -40,10 +40,12 @@ public class DiscordClient
// Don't validate because the token can have special characters // Don't validate because the token can have special characters
// https://github.com/Tyrrrz/DiscordChatExporter/issues/828 // https://github.com/Tyrrrz/DiscordChatExporter/issues/828
request.Headers.TryAddWithoutValidation( request
"Authorization", .Headers
tokenKind == TokenKind.Bot ? $"Bot {_token}" : _token .TryAddWithoutValidation(
); "Authorization",
tokenKind == TokenKind.Bot ? $"Bot {_token}" : _token
);
var response = await Http.Client.SendAsync( var response = await Http.Client.SendAsync(
request, request,
@@ -58,11 +60,13 @@ public class DiscordClient
// require properly keeping track of Discord's global/per-route/per-resource // require properly keeping track of Discord's global/per-route/per-resource
// rate limits and that's just way too much effort. // rate limits and that's just way too much effort.
// https://discord.com/developers/docs/topics/rate-limits // https://discord.com/developers/docs/topics/rate-limits
var remainingRequestCount = response.Headers var remainingRequestCount = response
.Headers
.TryGetValue("X-RateLimit-Remaining") .TryGetValue("X-RateLimit-Remaining")
?.Pipe(s => int.Parse(s, CultureInfo.InvariantCulture)); ?.Pipe(s => int.Parse(s, CultureInfo.InvariantCulture));
var resetAfterDelay = response.Headers var resetAfterDelay = response
.Headers
.TryGetValue("X-RateLimit-Reset-After") .TryGetValue("X-RateLimit-Reset-After")
?.Pipe(s => double.Parse(s, CultureInfo.InvariantCulture)) ?.Pipe(s => double.Parse(s, CultureInfo.InvariantCulture))
.Pipe(TimeSpan.FromSeconds); .Pipe(TimeSpan.FromSeconds);
@@ -0,0 +1,60 @@
using System;
using System.Collections.Generic;
using System.IO.Compression;
using System.Text.Json;
using System.Threading;
using System.Threading.Tasks;
using JsonExtensions.Reading;
namespace DiscordChatExporter.Core.Discord.Dump;
public partial class DataDump
{
public IReadOnlyList<DataDumpChannel> Channels { get; }
public DataDump(IReadOnlyList<DataDumpChannel> channels) => Channels = channels;
}
public partial class DataDump
{
public static DataDump Parse(JsonElement json)
{
var channels = new List<DataDumpChannel>();
foreach (var property in json.EnumerateObjectOrEmpty())
{
var channelId = Snowflake.Parse(property.Name);
var channelName = property.Value.GetString();
// Null items refer to deleted channels
if (channelName is null)
continue;
var channel = new DataDumpChannel(channelId, channelName);
channels.Add(channel);
}
return new DataDump(channels);
}
public static async ValueTask<DataDump> LoadAsync(
string zipFilePath,
CancellationToken cancellationToken = default
)
{
using var archive = ZipFile.OpenRead(zipFilePath);
var entry = archive.GetEntry("messages/index.json");
if (entry is null)
{
throw new InvalidOperationException(
"Could not find the channel index inside the data package."
);
}
await using var stream = entry.Open();
using var document = await JsonDocument.ParseAsync(stream, default, cancellationToken);
return Parse(document.RootElement);
}
}
@@ -0,0 +1,3 @@
namespace DiscordChatExporter.Core.Discord.Dump;
public record DataDumpChannel(Snowflake Id, string Name);
@@ -1,15 +1,15 @@
<Project Sdk="Microsoft.NET.Sdk"> <Project Sdk="Microsoft.NET.Sdk">
<ItemGroup> <ItemGroup>
<PackageReference Include="AsyncKeyedLock" Version="6.2.1" /> <PackageReference Include="AsyncKeyedLock" Version="6.2.2" />
<PackageReference Include="CSharpier.MsBuild" Version="0.25.0" PrivateAssets="all" /> <PackageReference Include="CSharpier.MsBuild" Version="0.26.1" PrivateAssets="all" />
<PackageReference Include="Gress" Version="2.1.1" /> <PackageReference Include="Gress" Version="2.1.1" />
<PackageReference Include="JsonExtensions" Version="1.2.0" /> <PackageReference Include="JsonExtensions" Version="1.2.0" />
<PackageReference Include="Polly" Version="8.0.0" /> <PackageReference Include="Polly" Version="8.1.0" />
<PackageReference Include="RazorBlade" Version="0.4.4" /> <PackageReference Include="RazorBlade" Version="0.4.4" />
<PackageReference Include="Superpower" Version="3.0.0" /> <PackageReference Include="Superpower" Version="3.0.0" />
<PackageReference Include="WebMarkupMin.Core" Version="2.14.0" /> <PackageReference Include="WebMarkupMin.Core" Version="2.14.0" />
<PackageReference Include="YoutubeExplode" Version="6.3.4" /> <PackageReference Include="YoutubeExplode" Version="6.3.7" />
</ItemGroup> </ItemGroup>
</Project> </Project>
@@ -22,11 +22,18 @@ public class ChannelExporter
{ {
// Forum channels don't have messages, they are just a list of threads // Forum channels don't have messages, they are just a list of threads
if (request.Channel.Kind == ChannelKind.GuildForum) if (request.Channel.Kind == ChannelKind.GuildForum)
throw new DiscordChatExporterException("Channel is a forum."); {
throw new DiscordChatExporterException(
"Channel is a forum and cannot be exported directly. "
+ "You need to pull its threads and export them individually."
);
}
// Check if the channel is empty // Check if the channel is empty
if (request.Channel.IsEmpty) if (request.Channel.IsEmpty)
{
throw new DiscordChatExporterException("Channel does not contain any messages."); throw new DiscordChatExporterException("Channel does not contain any messages.");
}
// Check if the 'after' boundary is valid // Check if the 'after' boundary is valid
if (request.After is not null && !request.Channel.MayHaveMessagesAfter(request.After.Value)) if (request.After is not null && !request.Channel.MayHaveMessagesAfter(request.After.Value))
@@ -88,7 +88,7 @@ internal partial class CsvMessageWriter : MessageWriter
await _writer.WriteAsync(','); await _writer.WriteAsync(',');
// Message timestamp // Message timestamp
await _writer.WriteAsync(CsvEncode(Context.FormatDate(message.Timestamp))); await _writer.WriteAsync(CsvEncode(Context.FormatDate(message.Timestamp, "o")));
await _writer.WriteAsync(','); await _writer.WriteAsync(',');
// Message content // Message content
@@ -7,6 +7,7 @@ using System.Text;
using System.Text.RegularExpressions; using System.Text.RegularExpressions;
using System.Threading; using System.Threading;
using System.Threading.Tasks; using System.Threading.Tasks;
using System.Web;
using AsyncKeyedLock; using AsyncKeyedLock;
using DiscordChatExporter.Core.Utils; using DiscordChatExporter.Core.Utils;
using DiscordChatExporter.Core.Utils.Extensions; using DiscordChatExporter.Core.Utils.Extensions;
@@ -64,7 +65,9 @@ internal partial class ExportAssetDownloader
// Try to set the file date according to the last-modified header // Try to set the file date according to the last-modified header
try try
{ {
var lastModified = response.Content.Headers var lastModified = response
.Content
.Headers
.TryGetValue("Last-Modified") .TryGetValue("Last-Modified")
?.Pipe( ?.Pipe(
s => s =>
@@ -101,12 +104,29 @@ internal partial class ExportAssetDownloader
internal partial class ExportAssetDownloader internal partial class ExportAssetDownloader
{ {
private static string GetUrlHash(string url) => private static string GetUrlHash(string url)
SHA256 {
.HashData(Encoding.UTF8.GetBytes(url)) // Remove signature parameters from Discord CDN URLs to normalize them
static string NormalizeUrl(string url)
{
var uri = new Uri(url);
if (!string.Equals(uri.Host, "cdn.discordapp.com", StringComparison.OrdinalIgnoreCase))
return url;
var query = HttpUtility.ParseQueryString(uri.Query);
query.Remove("ex");
query.Remove("is");
query.Remove("hm");
return uri.GetLeftPart(UriPartial.Path) + query;
}
return SHA256
.HashData(Encoding.UTF8.GetBytes(NormalizeUrl(url)))
.ToHex() .ToHex()
// 5 chars ought to be enough for anybody // 5 chars ought to be enough for anybody
.Truncate(5); .Truncate(5);
}
private static string GetFileNameFromUrl(string url) private static string GetFileNameFromUrl(string url)
{ {
@@ -101,7 +101,8 @@ internal class ExportContext
public Role? TryGetRole(Snowflake id) => _rolesById.GetValueOrDefault(id); public Role? TryGetRole(Snowflake id) => _rolesById.GetValueOrDefault(id);
public IReadOnlyList<Role> GetUserRoles(Snowflake id) => public IReadOnlyList<Role> GetUserRoles(Snowflake id) =>
TryGetMember(id)?.RoleIds TryGetMember(id)
?.RoleIds
.Select(TryGetRole) .Select(TryGetRole)
.WhereNotNull() .WhereNotNull()
.OrderByDescending(r => r.Position) .OrderByDescending(r => r.Position)
@@ -182,10 +182,9 @@ public partial class ExportRequest
=> before?.ToDate().ToString("yyyy-MM-dd", CultureInfo.InvariantCulture) => before?.ToDate().ToString("yyyy-MM-dd", CultureInfo.InvariantCulture)
?? "", ?? "",
"%d" "%d"
=> DateTimeOffset.Now.ToString( => DateTimeOffset
"yyyy-MM-dd", .Now
CultureInfo.InvariantCulture .ToString("yyyy-MM-dd", CultureInfo.InvariantCulture),
),
"%%" => "%", "%%" => "%",
_ => m.Value _ => m.Value
@@ -26,12 +26,14 @@ internal class ContainsMessageFilter : MessageFilter
public override bool IsMatch(Message message) => public override bool IsMatch(Message message) =>
IsMatch(message.Content) IsMatch(message.Content)
|| message.Embeds.Any( || message
e => .Embeds
IsMatch(e.Title) .Any(
|| IsMatch(e.Author?.Name) e =>
|| IsMatch(e.Description) IsMatch(e.Title)
|| IsMatch(e.Footer?.Text) || IsMatch(e.Author?.Name)
|| e.Fields.Any(f => IsMatch(f.Name) || IsMatch(f.Value)) || IsMatch(e.Description)
); || IsMatch(e.Footer?.Text)
|| e.Fields.Any(f => IsMatch(f.Name) || IsMatch(f.Value))
);
} }
@@ -11,11 +11,13 @@ internal class MentionsMessageFilter : MessageFilter
public MentionsMessageFilter(string value) => _value = value; public MentionsMessageFilter(string value) => _value = value;
public override bool IsMatch(Message message) => public override bool IsMatch(Message message) =>
message.MentionedUsers.Any( message
user => .MentionedUsers
string.Equals(_value, user.Name, StringComparison.OrdinalIgnoreCase) .Any(
|| string.Equals(_value, user.DisplayName, StringComparison.OrdinalIgnoreCase) user =>
|| string.Equals(_value, user.FullName, StringComparison.OrdinalIgnoreCase) string.Equals(_value, user.Name, StringComparison.OrdinalIgnoreCase)
|| string.Equals(_value, user.Id.ToString(), StringComparison.OrdinalIgnoreCase) || string.Equals(_value, user.DisplayName, StringComparison.OrdinalIgnoreCase)
); || string.Equals(_value, user.FullName, StringComparison.OrdinalIgnoreCase)
|| string.Equals(_value, user.Id.ToString(), StringComparison.OrdinalIgnoreCase)
);
} }
@@ -11,10 +11,16 @@ internal class ReactionMessageFilter : MessageFilter
public ReactionMessageFilter(string value) => _value = value; public ReactionMessageFilter(string value) => _value = value;
public override bool IsMatch(Message message) => public override bool IsMatch(Message message) =>
message.Reactions.Any( message
r => .Reactions
string.Equals(_value, r.Emoji.Id?.ToString(), StringComparison.OrdinalIgnoreCase) .Any(
|| string.Equals(_value, r.Emoji.Name, StringComparison.OrdinalIgnoreCase) r =>
|| string.Equals(_value, r.Emoji.Code, StringComparison.OrdinalIgnoreCase) string.Equals(
); _value,
r.Emoji.Id?.ToString(),
StringComparison.OrdinalIgnoreCase
)
|| string.Equals(_value, r.Emoji.Name, StringComparison.OrdinalIgnoreCase)
|| string.Equals(_value, r.Emoji.Code, StringComparison.OrdinalIgnoreCase)
);
} }
@@ -196,7 +196,8 @@ internal partial class HtmlMarkdownVisitor : MarkdownVisitor
// Try to extract the message ID if the link points to a Discord message // Try to extract the message ID if the link points to a Discord message
var linkedMessageId = Regex var linkedMessageId = Regex
.Match(link.Url, @"^https?://(?:discord|discordapp)\.com/channels/.*?/(\d+)/?$") .Match(link.Url, @"^https?://(?:discord|discordapp)\.com/channels/.*?/(\d+)/?$")
.Groups[1].Value; .Groups[1]
.Value;
_buffer.Append( _buffer.Append(
!string.IsNullOrWhiteSpace(linkedMessageId) !string.IsNullOrWhiteSpace(linkedMessageId)
@@ -143,6 +143,31 @@ internal class JsonMessageWriter : MessageWriter
await _writer.FlushAsync(cancellationToken); await _writer.FlushAsync(cancellationToken);
} }
private async ValueTask WriteEmbedVideoAsync(
EmbedVideo embedVideo,
CancellationToken cancellationToken = default
)
{
_writer.WriteStartObject();
if (!string.IsNullOrWhiteSpace(embedVideo.Url))
{
_writer.WriteString(
"url",
await Context.ResolveAssetUrlAsync(
embedVideo.ProxyUrl ?? embedVideo.Url,
cancellationToken
)
);
}
_writer.WriteNumber("width", embedVideo.Width);
_writer.WriteNumber("height", embedVideo.Height);
_writer.WriteEndObject();
await _writer.FlushAsync(cancellationToken);
}
private async ValueTask WriteEmbedFooterAsync( private async ValueTask WriteEmbedFooterAsync(
EmbedFooter embedFooter, EmbedFooter embedFooter,
CancellationToken cancellationToken = default CancellationToken cancellationToken = default
@@ -224,6 +249,12 @@ internal class JsonMessageWriter : MessageWriter
await WriteEmbedImageAsync(embed.Image, cancellationToken); await WriteEmbedImageAsync(embed.Image, cancellationToken);
} }
if (embed.Video is not null)
{
_writer.WritePropertyName("video");
await WriteEmbedVideoAsync(embed.Video, cancellationToken);
}
if (embed.Footer is not null) if (embed.Footer is not null)
{ {
_writer.WritePropertyName("footer"); _writer.WritePropertyName("footer");
@@ -419,12 +450,14 @@ internal class JsonMessageWriter : MessageWriter
_writer.WriteStartArray("users"); _writer.WriteStartArray("users");
await foreach ( await foreach (
var user in Context.Discord.GetMessageReactionsAsync( var user in Context
Context.Request.Channel.Id, .Discord
message.Id, .GetMessageReactionsAsync(
reaction.Emoji, Context.Request.Channel.Id,
cancellationToken message.Id,
) reaction.Emoji,
cancellationToken
)
) )
{ {
_writer.WriteStartObject(); _writer.WriteStartObject();
@@ -44,10 +44,10 @@ internal partial class MessageExporter : IAsyncDisposable
// Ensure that the partition limit has not been reached // Ensure that the partition limit has not been reached
if ( if (
_writer is not null _writer is not null
&& _context.Request.PartitionLimit.IsReached( && _context
_writer.MessagesWritten, .Request
_writer.BytesWritten .PartitionLimit
) .IsReached(_writer.MessagesWritten, _writer.BytesWritten)
) )
{ {
await ResetWriterAsync(cancellationToken); await ResetWriterAsync(cancellationToken);
@@ -30,7 +30,8 @@ internal class AggregateMatcher<T> : IMatcher<T>
// If this match is earlier than previous earliest - replace // If this match is earlier than previous earliest - replace
if ( if (
earliestMatch is null || match.Segment.StartIndex < earliestMatch.Segment.StartIndex earliestMatch is null
|| match.Segment.StartIndex < earliestMatch.Segment.StartIndex
) )
earliestMatch = match; earliestMatch = match;
@@ -24,12 +24,10 @@ internal class StringMatcher<T> : IMatcher<T>
public ParsedMatch<T>? TryMatch(StringSegment segment) public ParsedMatch<T>? TryMatch(StringSegment segment)
{ {
var index = segment.Source.IndexOf( var index = segment
_needle, .Source
segment.StartIndex, .IndexOf(_needle, segment.StartIndex, segment.Length, _comparison);
segment.Length,
_comparison
);
if (index < 0) if (index < 0)
return null; return null;
-1
View File
@@ -3,7 +3,6 @@ using DiscordChatExporter.Gui.ViewModels;
using DiscordChatExporter.Gui.ViewModels.Framework; using DiscordChatExporter.Gui.ViewModels.Framework;
using Stylet; using Stylet;
using StyletIoC; using StyletIoC;
#if !DEBUG #if !DEBUG
using System.Windows; using System.Windows;
using System.Windows.Threading; using System.Windows.Threading;
@@ -13,14 +13,14 @@
</ItemGroup> </ItemGroup>
<ItemGroup> <ItemGroup>
<PackageReference Include="Cogwheel" Version="2.0.3" /> <PackageReference Include="Cogwheel" Version="2.0.4" />
<PackageReference Include="CSharpier.MsBuild" Version="0.25.0" PrivateAssets="all" /> <PackageReference Include="CSharpier.MsBuild" Version="0.26.1" PrivateAssets="all" />
<PackageReference Include="Deorcify" Version="1.0.2" PrivateAssets="all" /> <PackageReference Include="Deorcify" Version="1.0.2" PrivateAssets="all" />
<PackageReference Include="DotnetRuntimeBootstrapper" Version="2.5.1" PrivateAssets="all" /> <PackageReference Include="DotnetRuntimeBootstrapper" Version="2.5.1" PrivateAssets="all" />
<PackageReference Include="Gress" Version="2.1.1" /> <PackageReference Include="Gress" Version="2.1.1" />
<PackageReference Include="MaterialDesignColors" Version="2.1.4" /> <PackageReference Include="MaterialDesignColors" Version="2.1.4" />
<PackageReference Include="MaterialDesignThemes" Version="4.9.0" /> <PackageReference Include="MaterialDesignThemes" Version="4.9.0" />
<PackageReference Include="Microsoft.Xaml.Behaviors.Wpf" Version="1.1.39" /> <PackageReference Include="Microsoft.Xaml.Behaviors.Wpf" Version="1.1.77" />
<PackageReference Include="Ookii.Dialogs.Wpf" Version="5.0.1" /> <PackageReference Include="Ookii.Dialogs.Wpf" Version="5.0.1" />
<PackageReference Include="Onova" Version="2.6.10" /> <PackageReference Include="Onova" Version="2.6.10" />
<PackageReference Include="PropertyChanged.Fody" Version="4.1.0" PrivateAssets="all" /> <PackageReference Include="PropertyChanged.Fody" Version="4.1.0" PrivateAssets="all" />
@@ -66,7 +66,8 @@ public partial class SettingsService
{ {
try try
{ {
return Registry.CurrentUser return Registry
.CurrentUser
.OpenSubKey( .OpenSubKey(
"SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Themes\\Personalize", "SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Themes\\Personalize",
false false
@@ -12,8 +12,8 @@ using DiscordChatExporter.Gui.Models;
using DiscordChatExporter.Gui.Services; using DiscordChatExporter.Gui.Services;
using DiscordChatExporter.Gui.Utils; using DiscordChatExporter.Gui.Utils;
using DiscordChatExporter.Gui.ViewModels.Dialogs; using DiscordChatExporter.Gui.ViewModels.Dialogs;
using DiscordChatExporter.Gui.ViewModels.Messages;
using DiscordChatExporter.Gui.ViewModels.Framework; using DiscordChatExporter.Gui.ViewModels.Framework;
using DiscordChatExporter.Gui.ViewModels.Messages;
using Gress; using Gress;
using Gress.Completable; using Gress.Completable;
using Stylet; using Stylet;
@@ -234,7 +234,8 @@ public class DashboardViewModel : PropertyChangedBase
var exporter = new ChannelExporter(_discord); var exporter = new ChannelExporter(_discord);
var channelProgressPairs = dialog.Channels! var channelProgressPairs = dialog
.Channels!
.Select(c => new { Channel = c, Progress = _progressMuxer.CreateInput() }) .Select(c => new { Channel = c, Progress = _progressMuxer.CreateInput() })
.ToArray(); .ToArray();
@@ -4,8 +4,8 @@ using DiscordChatExporter.Gui.Services;
using DiscordChatExporter.Gui.Utils; using DiscordChatExporter.Gui.Utils;
using DiscordChatExporter.Gui.ViewModels.Components; using DiscordChatExporter.Gui.ViewModels.Components;
using DiscordChatExporter.Gui.ViewModels.Dialogs; using DiscordChatExporter.Gui.ViewModels.Dialogs;
using DiscordChatExporter.Gui.ViewModels.Messages;
using DiscordChatExporter.Gui.ViewModels.Framework; using DiscordChatExporter.Gui.ViewModels.Framework;
using DiscordChatExporter.Gui.ViewModels.Messages;
using MaterialDesignThemes.Wpf; using MaterialDesignThemes.Wpf;
using Stylet; using Stylet;
+5 -4
View File
@@ -23,7 +23,8 @@
**DiscordChatExporter** is an application that can be used to export message history from any [Discord](https://discord.com) channel to a file. **DiscordChatExporter** is an application that can be used to export message history from any [Discord](https://discord.com) channel to a file.
It works with direct messages, group messages, and server channels, and supports Discord's dialect of markdown as well as most other rich media features. It works with direct messages, group messages, and server channels, and supports Discord's dialect of markdown as well as most other rich media features.
> ❔ If you have questions or issues, **please refer to the [docs](.docs)**. <!-- Can't use a relative link here due to a bug in markdown parsing -->
> ❔ If you have questions or issues, **please refer to the [docs](https://github.com/Tyrrrz/DiscordChatExporter/tree/master/.docs)**.
> 💬 If you want to chat, **join my [Discord server](https://discord.gg/2SUWKFnHSm)**. > 💬 If you want to chat, **join my [Discord server](https://discord.gg/2SUWKFnHSm)**.
@@ -57,7 +58,7 @@ The following table lists all available download options:
<td> <td>
<ul> <ul>
<li>🟢 <b><a href="https://github.com/Tyrrrz/DiscordChatExporter/releases/latest">Stable release</a></b> (<code>DiscordChatExporter.zip</code>)</li> <li>🟢 <b><a href="https://github.com/Tyrrrz/DiscordChatExporter/releases/latest">Stable release</a></b> (<code>DiscordChatExporter.zip</code>)</li>
<li>🟠 <a href="https://github.com/Tyrrrz/DiscordChatExporter/actions/workflows/main.yml">CI build</a> (<code>DiscordChatExporter.zip</code>)</li> <li>🟠 <a href="https://github.com/Tyrrrz/DiscordChatExporter/actions/workflows/main.yml">CI build</a> (<code>DiscordChatExporter.Gui.zip</code>)</li>
</ul> </ul>
</td> </td>
<td> <td>
@@ -70,8 +71,8 @@ The following table lists all available download options:
<td><b>CLI</b></td> <td><b>CLI</b></td>
<td> <td>
<ul> <ul>
<li>🟢 <b><a href="https://github.com/Tyrrrz/DiscordChatExporter/releases/latest">Stable release</a></b> (<code>DiscordChatExporter.CLI.zip</code></li> <li>🟢 <b><a href="https://github.com/Tyrrrz/DiscordChatExporter/releases/latest">Stable release</a></b> (<code>DiscordChatExporter.Cli.zip</code></li>
<li>🟠 <a href="https://github.com/Tyrrrz/DiscordChatExporter/actions/workflows/main.yml">CI build</a> (<code>DiscordChatExporter.CLI.zip</code>)</li> <li>🟠 <a href="https://github.com/Tyrrrz/DiscordChatExporter/actions/workflows/main.yml">CI build</a> (<code>DiscordChatExporter.Cli.zip</code>)</li>
<li>🐋 <a href="https://hub.docker.com/r/tyrrrz/discordchatexporter">Docker</a> (<code>tyrrrz/discordchatexporter</code>)</li> <li>🐋 <a href="https://hub.docker.com/r/tyrrrz/discordchatexporter">Docker</a> (<code>tyrrrz/discordchatexporter</code>)</li>
<li>📦 <a href="https://aur.archlinux.org/packages/discord-chat-exporter-cli">AUR</a> (<code>discord-chat-exporter-cli</code>)</li> <li>📦 <a href="https://aur.archlinux.org/packages/discord-chat-exporter-cli">AUR</a> (<code>discord-chat-exporter-cli</code>)</li>
<li>📦 <a href="https://search.nixos.org/packages?query=discordchatexporter-cli">Nix</a> (<code>discordchatexporter-cli</code>)</li> <li>📦 <a href="https://search.nixos.org/packages?query=discordchatexporter-cli">Nix</a> (<code>discordchatexporter-cli</code>)</li>