Compare commits

...

13 Commits

Author SHA1 Message Date
Jann Stute 119bcc5f82 fix: add empty library fixture for version 202 2026-07-13 11:31:59 +02:00
Jann Stute f1503f75fb fix: bump db version to 300, since it is a breaking change 2026-07-13 11:21:52 +02:00
Jann Stute d7b2a5efa7 fix: missing filename column after migration 2026-07-13 11:17:45 +02:00
Jann Stute ab3b1f2f17 fix: remove folder param from all uses of Entry constructor 2026-07-12 13:11:44 +02:00
Jann Stute 5e1d4a1190 Merge branch 'main' into refactor-remove-folders 2026-07-12 13:05:18 +02:00
Xarvex aa2d9d4815 fix(ci): misapplied rebase, cont 9d5200b
fixes: #1445 (for good)
2026-07-11 19:33:29 -05:00
Xarvex 16cfa8d2ff chore(ci): ignore translations pushes for reuse 2026-07-11 19:27:21 -05:00
Xarvex 9d5200b2f2 fix(ci): path matching fixes, corrected concurrency
Pull requests are now correctly handled and concurrency is per-job.
translations branch can safely be ignored for pushes.
2026-07-11 19:19:42 -05:00
Jann Stute 69dce93af5 refactor: remove dead folders table 2026-07-11 23:41:39 +02:00
Jann Stute f252a86fd5 refactor: split out sql migrations from library (#1432)
* refactor: minor simplification

* refactor: split open_library into new and not new case

Functionality is entirely unchanged, but entire method is duplicated.
Removing dead code from either copy is next.

* refactor: remove dead code

* doc: add todos and remove trivially true assert

* refactor: add assurance 1 version check

Assurance 1 was introduced with library version 101, specifically commmit 12e074b71d.
Thus all libraries created on version 101 and above already fullfill it and don't need it.
Furthermore, it only fails if the library didn't need the assurance, meaning the try-except and warning catching can be removed

* refactor: remove various unnecessary try-except statements in new_lib

All of these were introduced to account for the differences between new libraries and existing ones,
which makes them unnecessary after this split.

* refactor: remove unnecessary check in new_lib

* refactor: move folder assurance after migrations

The folder assurance has been present since the very first SQL commit e5e7b8afc6,
and thus only has an effect when the library is moved, meaning that is semantically not part of the migrations.

* refactor: massively simplify open_library

* refactor: move engine creation to static method

* refactor: add version check for assurance 3

Assurance 3 was introduced in commit 47c3d5338f
shortly (24h 20min) after the DB version had been bumped to 200.
Since it was also included in the first release with DB version 200,
all libraries created on DB version 200 and above
can be presumed to have already had this applied on creation.

* refactor: add assurance 3 to DB 200 migration

* refactor: move assurance 1 to a proper migration method

Moving the assurance after the migrations 7, 8, 9, and 100 is fine because it doesn't affect those.

* refactor: update version after every successfull migration

Since every migration migrates the library to a certain DB_VERSION, we can simply update the library version after such a successfull migration.
This way if a migration fails, the previous migrations won't be rerun the next time the library is opened.

* refactor: rewrite migration procedure as loop

* refactor: apply migration and update version in same transaction

None of the content of the migrations is changed, only the `with session:` statements are removed.

* refactor: replace all commits in the migrations with flushes

Also removes various try-except statements in the migrations.
These were introduced to catch the case where the migration is run twice due to a later migration failing,
this is not necessary anymore as every migration will only be executed at most once per library.

* refactor: make sure the migration log statements are consistent

* fix: pass library dir to migrations

* fix(db migration 8): only add colors that are actually new

DB Migration 8 was previously adding all colors except for the neon ones,
however, all but three of those have been around since DB version 4,
meaning that they don't need to be added at all (which caused the tests to fail).

* fix: json migration used outdated interface

* fix(open_library): create TS directory only if not opened in memory

* fix: enable sane transaction behaviour

By default in SQLAlchemy schema changes are automatically committed,
even if auto-commit is turned off.
This commit turns off auto-commit completely,
which caused some schema to not be applied correctly,
but those were fixed here as well.

* refactor: hide 'argument is not accessed' notices

* fix(db migration 9): filename property wasn't written correctly

* fix(db migration 104): include/exclude list was loaded incorrectly

* feat: log start and end of DB migrations

* fix: don't use double transaction in open_sqlite_library

* fix: only commit once when creating new library

* doc: add comment on removing Folder logic

* refactor: remove dunder naming
2026-07-11 12:36:55 -07:00
Xarvex a0fb679729 fix(ci): more path matching 2026-07-11 01:29:11 -05:00
Xarvex b182b2ff7e chore(ci): more paths for pytest to trigger on 2026-07-09 22:23:17 -05:00
purpletennisball 308b36b31e fix: thumbnail rendering for older pxd files (#1441) 2026-07-09 18:03:47 -07:00
14 changed files with 536 additions and 525 deletions
+64 -12
View File
@@ -8,26 +8,29 @@ on:
paths: &on_paths
- .github/actions/setup-python/action.yml
- .github/workflows/checks_python.yml
- src/**/resources/**
- src/**/*.json
- src/**/*.qrc
- tests/**
- .editorconfig
- pyproject.toml
- uv.lock
- '**.py'
- '**.pyi'
- 'src/tagstudio/resources/**'
- '**.pyi?'
push:
branches-ignore:
- translations
paths: *on_paths
workflow_dispatch:
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
defaults:
run:
shell: sh
jobs:
run-conditions:
permissions:
pull-requests: read
name: Run Conditions
runs-on: ubuntu-latest
outputs:
@@ -36,14 +39,25 @@ jobs:
ruff: ${{ steps.run-conditions.outputs.ruff }}
steps:
- name: Checkout repository
if: github.event_name != 'pull_request'
uses: actions/checkout@v7
with:
fetch-depth: 0
# Largest positive number; infinite depth.
# Using 0 would grab all branches.
# See: https://github.com/actions/checkout/issues/520
# See: https://stackoverflow.com/questions/6802145/how-to-convert-a-git-shallow-clone-to-a-full-clone/6802238#6802238
# `git fetch --unshallow` as suggested in later answers would be an extra operation.
fetch-depth: '2147483647'
- name: Check changed files
id: changed-files
uses: tj-actions/changed-files@v47.0.6
with:
fail_on_initial_diff_error: 'true'
fail_on_submodule_diff_error: 'true'
skip_initial_fetch: 'true'
# WARNING: Does not support `?` glob operand!
files_yaml: |
generic:
- .github/workflows/checks_python.yml
@@ -55,20 +69,51 @@ jobs:
- .github/actions/setup-python/action.yml
pytest:
- .github/actions/setup-python/action.yml
- 'src/tagstudio/resources/**'
- src/**/resources/**
- src/**/*.json
- src/**/*.qrc
- tests/**
ruff:
- .editorconfig
- name: Set run conditions
id: run-conditions
env:
CHANGED_GENERIC: ${{ steps.changed-files.outputs.generic_any_changed }}
CHANGED_PYRIGHT: ${{ steps.changed-files.outputs.pyright_any_changed }}
CHANGED_PYTEST: ${{ steps.changed-files.outputs.pytest_any_changed }}
CHANGED_RUFF: ${{ steps.changed-files.outputs.ruff_any_changed }}
run: |
pyright=false
pytest=false
ruff=false
if [ "${CHANGED_GENERIC}" = true ]; then
pyright=true
pytest=true
ruff=true
else
if [ "${CHANGED_PYRIGHT}" = true ]; then
pyright=true
fi
if [ "${CHANGED_PYTEST}" = true ]; then
pytest=true
fi
if [ "${CHANGED_RUFF}" = true ]; then
ruff=true
fi
fi
cat <<EOF >>"${GITHUB_OUTPUT}"
pyright=${{ steps.changed-files.outputs.generic_any_changed || steps.changed-files.outputs.pyright_any_changed }}
pytest=${{ steps.changed-files.outputs.generic_any_changed || steps.changed-files.outputs.pytest_any_changed }}
ruff=${{ steps.changed-files.outputs.generic_any_changed || steps.changed-files.outputs.ruff_any_changed }}
pyright=${pyright}
pytest=${pytest}
ruff=${ruff}
EOF
check-pyright:
concurrency:
group: pyright-${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
name: Pyright
needs: run-conditions
if: needs.run-conditions.outputs.pyright == 'true'
@@ -89,6 +134,9 @@ jobs:
run: pyright
check-pytest:
concurrency:
group: ${{ matrix.os }}-pytest-${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
strategy:
fail-fast: false
matrix:
@@ -151,6 +199,10 @@ jobs:
run: pytest
check-ruff:
concurrency:
group: ruff-${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
name: Ruff
needs: run-conditions
if: needs.run-conditions.outputs.ruff == 'true'
+6 -1
View File
@@ -3,7 +3,12 @@
---
name: REUSE Compliance Check
on: [pull_request, push]
on:
pull_request:
push:
branches-ignore:
- translations
workflow_dispatch:
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
@@ -9,7 +9,7 @@ JSON_FILENAME: str = "ts_library.json"
DB_VERSION_CURRENT_KEY: str = "CURRENT"
DB_VERSION_INITIAL_KEY: str = "INITIAL"
DB_VERSION: int = 202
DB_VERSION: int = 300
TAG_CHILDREN_QUERY = text("""
WITH RECURSIVE ChildTags AS (
+10 -6
View File
@@ -42,13 +42,17 @@ def make_engine(connection_string: str) -> Engine:
def make_tables(engine: Engine) -> None:
logger.info("[Library] Creating DB tables...")
Base.metadata.create_all(engine)
# tag IDs < 1000 are reserved
# create tag and delete it to bump the autoincrement sequence
# TODO - find a better way
# is this the better way?
with engine.connect() as conn:
# TODO: this should instead be migrations that create the exact tables that were added in
# the respective DB versions
Base.metadata.create_all(conn)
conn.commit()
# TODO: this needs to be a migration
# tag IDs < 1000 are reserved
# create tag and delete it to bump the autoincrement sequence
# TODO - find a better way
# is this the better way?
result = conn.execute(text("SELECT SEQ FROM sqlite_sequence WHERE name='tags'"))
autoincrement_val = result.scalar()
if not autoincrement_val or autoincrement_val <= RESERVED_TAG_END:
File diff suppressed because it is too large Load Diff
@@ -182,23 +182,11 @@ class Tag(Base):
return self.name >= other.name
class Folder(Base):
__tablename__ = "folders"
# TODO - implement this
id: Mapped[int] = mapped_column(primary_key=True)
path: Mapped[Path] = mapped_column(PathType, unique=True)
uuid: Mapped[str] = mapped_column(unique=True)
class Entry(Base):
__tablename__ = "entries"
id: Mapped[int] = mapped_column(primary_key=True)
folder_id: Mapped[int] = mapped_column(ForeignKey("folders.id"))
folder: Mapped[Folder] = relationship("Folder")
path: Mapped[Path] = mapped_column(PathType, unique=True)
filename: Mapped[str] = mapped_column()
suffix: Mapped[str] = mapped_column()
@@ -235,7 +223,6 @@ class Entry(Base):
def __init__(
self,
path: Path,
folder: Folder,
fields: list[BaseField],
id: int | None = None,
date_created: dt | None = None,
@@ -244,7 +231,6 @@ class Entry(Base):
) -> None:
super().__init__()
self.path = path
self.folder = folder
self.id = id # pyright: ignore[reportAttributeAccessIssue]
self.filename = path.name
self.suffix = path.suffix.lstrip(".").lower()
-2
View File
@@ -16,7 +16,6 @@ from tagstudio.core.library.alchemy.library import Library
from tagstudio.core.library.alchemy.models import Entry
from tagstudio.core.library.ignore import PATH_GLOB_FLAGS, Ignore, ignore_to_glob
from tagstudio.core.utils.silent_subprocess import silent_run # pyright: ignore
from tagstudio.core.utils.types import unwrap
logger = structlog.get_logger(__name__)
@@ -41,7 +40,6 @@ class RefreshTracker:
entries = [
Entry(
path=entry_path,
folder=unwrap(self.library.folder),
fields=[],
date_added=dt.now(),
)
+6 -5
View File
@@ -395,14 +395,15 @@ class JsonMigrationModal(QObject):
# Convert JSON Library to SQLite
yield Translations["json_migration.creating_database_tables"]
self.sql_lib = SqliteLibrary()
self.temp_path: Path = (
self.json_lib.library_dir / TS_FOLDER_NAME / "migration_ts_library.sqlite"
)
temp_filename = "migration_ts_library.sqlite"
self.temp_path: Path = self.json_lib.library_dir / TS_FOLDER_NAME / temp_filename
if self.temp_path.exists():
logger.info('Temporary migration file "temp_path" already exists. Removing...')
self.temp_path.unlink()
self.sql_lib.open_sqlite_library(
self.json_lib.library_dir, is_new=True, storage_path=str(self.temp_path)
self.sql_lib.create_sqlite_library(
self.json_lib.library_dir,
in_memory=False,
sql_filename=temp_filename,
)
yield Translations.format(
"json_migration.migrating_files_entries", entries=len(self.json_lib.entries)
+2
View File
@@ -1229,6 +1229,8 @@ class ThumbRenderer(QObject):
"QuickLook/Preview.heic",
"QuickLook/Thumbnail.jpg",
"QuickLook/Thumbnail.heic",
"QuickLook/Thumbnail.webp",
"QuickLook/Icon.webp",
]
im: Image.Image | None = None
-8
View File
@@ -20,7 +20,6 @@ sys.path.insert(0, str(CWD.parent))
from tagstudio.core.constants import THUMB_CACHE_NAME, TS_FOLDER_NAME
from tagstudio.core.library.alchemy.library import Library
from tagstudio.core.library.alchemy.models import Entry, Tag
from tagstudio.core.utils.types import unwrap
from tagstudio.qt.thumb_grid_layout import ThumbGridLayout
from tagstudio.qt.ts_qt import QtDriver
@@ -36,22 +35,18 @@ def file_mediatypes_library():
status = lib.open_library(Path(""), in_memory=True)
assert status.success
folder = unwrap(lib.folder)
entry1 = Entry(
folder=folder,
path=Path("foo.png"),
fields=[TextField(name="Title", value="I'm a Test Title")],
)
entry2 = Entry(
folder=folder,
path=Path("bar.png"),
fields=[TextField(name="Title", value="I'm a Test Title")],
)
entry3 = Entry(
folder=folder,
path=Path("baz.apng"),
fields=[TextField(name="Title", value="I'm a Test Title")],
)
@@ -87,7 +82,6 @@ def library(request, library_dir: Path): # pyright: ignore
lib = Library()
status = lib.open_library(library_path, in_memory=True)
assert status.success
folder = unwrap(lib.folder)
tag = Tag(
name="foo",
@@ -116,7 +110,6 @@ def library(request, library_dir: Path): # pyright: ignore
# default item with deterministic name
entry = Entry(
id=1,
folder=folder,
path=Path("foo.txt"),
fields=[TextField(name="Title", value="I'm a Test Title")],
)
@@ -124,7 +117,6 @@ def library(request, library_dir: Path): # pyright: ignore
entry2 = Entry(
id=2,
folder=folder,
path=Path("one/two/bar.md"),
fields=[TextField(name="Title", value="I'm a Test Title")],
)
Binary file not shown.
-4
View File
@@ -8,25 +8,21 @@ from tagstudio.core.library.alchemy.fields import BaseField, TextField
from tagstudio.core.library.alchemy.library import Library
from tagstudio.core.library.alchemy.models import Entry
from tagstudio.core.library.alchemy.registries.dupe_files_registry import DupeFilesRegistry
from tagstudio.core.utils.types import unwrap
CWD = Path(__file__).parent
def test_refresh_dupe_files(library: Library):
library.library_dir = Path("/tmp/")
folder = unwrap(library.folder)
fields: list[BaseField] = [TextField(name="Title", value="I'm a Test Title")]
entry = Entry(
folder=folder,
path=Path("bar/foo.txt"),
fields=fields,
)
entry2 = Entry(
folder=folder,
path=Path("foo/foo.txt"),
fields=fields,
)
+1 -10
View File
@@ -77,7 +77,6 @@ def test_library_add_file(library: Library):
"""Check Entry.path handling for insert vs lookup"""
entry = Entry(
path=Path("bar.txt"),
folder=unwrap(library.folder),
fields=[TextField(name="Title", value="I'm a Test Title")],
)
@@ -139,8 +138,7 @@ def test_get_entry(library: Library, entry_min: Entry):
def test_entries_count(library: Library):
folder = unwrap(library.folder)
entries = [Entry(path=Path(f"{x}.txt"), folder=folder, fields=[]) for x in range(10)]
entries = [Entry(path=Path(f"{x}.txt"), fields=[]) for x in range(10)]
new_ids = library.add_entries(entries)
assert len(new_ids) == 10
@@ -254,7 +252,6 @@ def test_update_entry_with_multiple_identical_text_fields(library: Library, entr
def test_mirror_entry_fields(library: Library):
# Create and add entries with fields
entry_a = Entry(
folder=unwrap(library.folder),
path=Path("title_and_date.txt"),
fields=[
TextField(name="Title", value="I'm a Test Title"),
@@ -262,7 +259,6 @@ def test_mirror_entry_fields(library: Library):
],
)
entry_b = Entry(
folder=unwrap(library.folder),
path=Path("notes.txt"),
fields=[
TextField(name="Notes", value="These are my notes.\nNo peeking!", is_multiline=True),
@@ -270,7 +266,6 @@ def test_mirror_entry_fields(library: Library):
],
)
entry_c = Entry(
folder=unwrap(library.folder),
path=Path("date_published.txt"),
fields=[
DatetimeField(name="Date Published", value="2000-01-01 12:00:00"),
@@ -319,14 +314,11 @@ def test_mirror_entry_fields(library: Library):
def test_merge_entries(library: Library):
folder = unwrap(library.folder)
tag_0: Tag = unwrap(library.add_tag(Tag(id=1010, name="tag_0")))
tag_1: Tag = unwrap(library.add_tag(Tag(id=1011, name="tag_1")))
tag_2: Tag = unwrap(library.add_tag(Tag(id=1012, name="tag_2")))
entry_a = Entry(
folder=folder,
path=Path("a"),
fields=[
TextField(name="Author", value="Author McAuthorson"),
@@ -334,7 +326,6 @@ def test_merge_entries(library: Library):
],
)
entry_b = Entry(
folder=folder,
path=Path("b"),
fields=[TextField(name="Notes", value="test note", is_multiline=True)],
)