2 Commits
Author SHA1 Message Date
Olaf 0055cc197e incomming mail processing 2026-09-30 11:53:12 +02:00
Olaf b89d12769c Privacy assesment by AI.
Build LinkLog Development Image / development-image (push) Successful in 28s
2026-09-30 09:08:52 +02:00
15 changed files with 713 additions and 129 deletions
+1
View File
@@ -1,6 +1,7 @@
# Changelog
## Version v0.3.1
### Features
* LinkLog can now poll a configured IMAP mailbox and turn emailed links from a user's verified account address into saved entries with parsed comments and tags
### Fixed
* Made the feed profile summary visually distinct with a contrasting background and slightly smaller text
### Modification
+45 -109
View File
@@ -1,148 +1,84 @@
# LinkLog Firefox Extension Privacy Validation
Date: 2026-09-15
# LinkLog Privacy and Fingerprinting
## Scope
This report validates the privacy-relevant properties of the Firefox extension in `webextension/` only. It does not treat the LinkLog backend as part of the extension, except where the extension explicitly sends data to the user-configured backend.
## Validation Method
- Reviewed the extension manifest, popup and options scripts, localized privacy text, and packaged-page HTML.
- Searched the extension source for storage, permission, tab-capture, cookie, history, clipboard, and network APIs.
- Ran `python3 scripts/release/validate_release.py` to confirm the checked-in signed XPI and Firefox disclosure metadata are consistent.
This document describes the privacy and fingerprinting characteristics of the LinkLog web service and Firefox extension as implemented in this repository. It is a source-code assessment, not a legal privacy policy, penetration test, or guarantee about a particular deployment.
## Summary
The extension's implemented privacy behavior is mostly consistent with its declared purpose:
LinkLog does not contain explicit canvas, WebGL, audio, font, hardware, timezone, analytics, or third-party advertising fingerprinting code. Its main privacy risk is different: LinkLog is designed to publish link activity, and that activity can form a highly distinctive identity profile.
- It does not include telemetry, analytics, adtech, crash reporting, or third-party beaconing.
- It captures website activity only from the active tab when the user opens the popup.
- It stores long-lived account metadata locally, but keeps bearer session credentials in Firefox session storage rather than persistent extension storage.
- It sends captured link data only to the user-selected backend, plus standard extension-update traffic to the configured self-update URL.
- Firefox privacy disclosure metadata is present and validated for the signed XPI.
A deployment operator, public visitor, upstream website, Mastodon instance, or network observer may be able to correlate a user through the data and request patterns described below.
The main residual privacy risk is that the extension accepts both `http` and `https` backend URLs. That is useful for local development, but a non-local HTTP backend would expose credentials and saved-link data in transit.
## Fingerprinting and Correlation Risks
## Verified Properties
### Public activity and identity profile
### 1. Declared Firefox permissions are narrow
The public feed exposes usernames, profile avatars, bios, exact creation timestamps, titles, original URLs, comments, tags, and whether an entry was posted to Mastodon. Public user enumeration and per-user feed URLs make it easy to collect this information for a particular account.
- Required permissions are limited to `activeTab` and `storage`.
- Host access is not pre-granted broadly at install time. Instead, the extension declares optional HTTP and HTTPS host permissions and requests access only for the specific backend origin entered by the user.
- The signed-release validator enforces the Firefox `data_collection_permissions` declaration of required `websiteActivity` and no optional data-collection categories.
A sequence of saved links, topics, tags, timestamps, writing style, and referenced websites can be distinctive enough to associate a LinkLog account with activity on other services. This is a high privacy risk for users who expect saved links to be private.
Assessment: consistent with a least-privilege model for a link-saving extension.
Relevant implementation: `backend/app/api/public.py` and `backend/app/services/link_service.py`.
### 2. Website activity access is limited to the active tab
### URL and query-parameter leakage
- The popup reads the current page title and URL via `browser.tabs.query({ active: true, currentWindow: true })`.
- No content scripts are injected into visited pages.
- No background/service-worker logic observes browsing continuously.
LinkLog removes a configurable list of common advertising and analytics parameters, including `utm_*`, `gclid`, `fbclid`, and several vendor-specific parameters. This reduces routine campaign tracking but does not make URLs anonymous.
Assessment: the extension collects the minimum page context needed for its stated function when the user explicitly opens the popup.
Other query parameters, path segments, fragments that are retained by the URL cleaner, document identifiers, search terms, access codes, repository names, and user-specific URLs may still identify a person or expose sensitive information. Operators should treat saved URLs, comments, titles, and tags as potentially personal data.
### 3. Local storage use is bounded and understandable
The tracking-parameter list is configured through `LINKLOG_TRACKING_PARAMS`; changing it can also change the URL-normalization behavior of a deployment.
Persistent local storage:
### Firefox extension website activity
- `backendUrl`
- `email`
- `username`
The extension uses `activeTab` and requests optional HTTP/HTTPS host permissions for the configured LinkLog backend. When a user saves a page, the extension reads the active page's URL and title and sends the selected data to that backend.
Session-scoped storage:
The extension therefore handles website activity by design. A malicious or compromised configured backend could receive the URLs that users submit, and a user can disclose sensitive page URLs by saving them. The extension stores the backend URL and username in local extension storage and keeps session credentials in Firefox session storage.
- `accessToken`
- `refreshToken`
- `tokenExpiresAt`
- `deviceId`
Relevant implementation: `webextension/manifest.json`, `webextension/popup.js`, and `webextension/options.js`.
Observed behavior:
### Mastodon and external-service correlation
- Access and refresh credentials are stored in `browser.storage.session`.
- Sign-out and token invalidation clear those session values and also remove older legacy token keys from `browser.storage.local`.
- Account-identifying metadata remains in persistent storage to preserve configuration across browser restarts.
When enabled, the Mastodon plugin publishes the link title, comment, tags, source URL, and a UTC timestamp to the configured Mastodon instance. Posts include a recognizable `User-Agent: LinkLog/1.0` in server-to-server requests. The resulting public Mastodon post can link the user's LinkLog identity, interests, and activity times to a Mastodon account.
Assessment: token persistence is minimized appropriately, but the extension still retains backend/account metadata locally until changed or removed.
The plugin also makes LinkLog activity observable to the configured Mastodon server, including the instance connection and publication time.
### 4. Outbound network traffic is limited and attributable
Relevant implementation: `backend/app/services/plugin_manager.py`.
User-configured backend traffic:
### Server-side URL fetching
- `POST /api/auth/login`
- `POST /api/auth/refresh`
- `POST /api/auth/logout`
- `GET /api/auth/me`
- `GET /api/tags`
- `GET /api/public/config`
- `GET /api/user/plugins/mastodon`
- `GET /api/links/check`
- `GET /api/scrape`
- `POST /api/links`
The authenticated scrape endpoint fetches a user-provided URL from the LinkLog server to obtain a page title. The target website may see the LinkLog server's network address and request characteristics rather than the user's browser address. This creates server-side attribution and may reveal that a URL was submitted to LinkLog.
What the extension sends to the backend:
### Deployment fingerprinting
- Sign-in credentials: email, password, optional OTP.
- Session tokens in `Authorization` headers after login.
- Link payloads: title, cleaned URL, comment, timestamp, and tags.
- Lookup values for duplicate detection and title scraping.
A deployment may be distinguishable through its public version, FastAPI/OpenAPI metadata, static asset version parameters, enabled themes, feed page sizes, maximum post length, response headers, health endpoint, public hostname, and extension update metadata. These signals generally identify an installation or software version rather than a person, but they can assist cross-site correlation and targeted attack reconnaissance.
Other network destinations:
The public configuration endpoint intentionally returns feed page sizes and the maximum post-character limit. The application also exposes `/health`, and the README documents `/docs` and OpenAPI access. Production operators should decide which of these should remain public.
- Firefox self-update metadata and signed XPI download are configured via `webextension/updates.json` and the manifest `update_url` at `git.kolkman.org`.
- Repository links in the popup and options page go to `git.kolkman.org` only when the user clicks them.
### Authentication and account-state observation
Assessment: no hidden third-party data sinks were found in the extension code.
Login throttling, response status differences, verification state, reset-mail behavior, response timing, and refresh-token behavior can reveal limited information about account state to a party able to make repeated requests. Current login errors are mostly generic, but verified and unverified account paths still differ, and failed login handling may trigger password-reset mail for known verified accounts when SMTP is configured.
### 5. The extension avoids higher-risk browser APIs
Client IP handling and throttling are also deployment-sensitive. In a multi-instance deployment, the current SQLite-based limiter does not provide a shared, atomic privacy or abuse-control boundary.
No use was found of:
## Recommended Mitigations
- cookies APIs
- browsing history APIs
- bookmarks APIs
- clipboard APIs
- native messaging
- remote script loading
- `XMLHttpRequest` or `sendBeacon`
Prioritize these changes for privacy-sensitive or public deployments:
Assessment: this reduces both privacy scope and accidental data leakage paths.
1. Make feeds and links private by default, with explicit per-link or per-profile publication controls.
2. Make public user enumeration and profile indexing opt-in, and consider requiring authentication for user lists and private feeds.
3. Strip or allow-list URL query parameters and warn users before saving URLs that contain credentials, access codes, search terms, or other sensitive identifiers.
4. Offer privacy controls for avatars, bios, tags, comments, exact timestamps, and original URLs; consider timestamp coarsening for public entries.
5. Minimize extension permissions and clearly disclose that saving a page sends its URL and title to the configured backend. Keep backend origin permissions restricted to the configured origin.
6. Make Mastodon publication an explicit opt-in, show the complete data that will be published, and allow users to disable URL and timestamp inclusion.
7. Protect or disable `/docs`, `/openapi.json`, detailed health/configuration endpoints, version disclosures, and unnecessary response metadata in production.
8. Use uniform authentication responses and timing where practical, independently throttle password-reset mail, and use a shared atomic rate limiter such as Redis for multi-instance deployments.
9. Configure strict security headers, trusted hosts, HTTPS, log retention, and access controls. Do not log authorization headers, tokens, passwords, OTP values, or secret-bearing URLs.
10. Document retention, deletion, backup, and export behavior for links, profiles, audit records, avatars, tokens, and server logs.
### 6. Packaged extension pages use a restrictive CSP
## User Guidance
- Extension pages declare `script-src 'self'; object-src 'none'`.
- Popup and options pages load only bundled local scripts and assets.
Do not save private document links, password-reset URLs, invitation URLs, access tokens, or URLs containing sensitive query parameters to a public feed. Review titles, comments, tags, timestamps, and the Mastodon preview before publishing. Use a private deployment and disable Mastodon integration when the link history itself is sensitive.
Assessment: this materially lowers the risk of third-party script injection into extension pages.
## Assessment Limits
### 7. The privacy notice is broadly accurate, with one caveat
The options page says the extension does not collect personal data beyond what is necessary to log links to the LinkLog server. That statement is broadly supported by the code, because the extension only stores and transmits data needed for authentication and link submission.
Caveat: the notice would be more precise if it explicitly mentioned that the extension stores the configured backend URL, email address, and username locally, and stores session credentials for the current browser session.
## Residual Privacy Risks
### R1. Non-TLS backends are allowed
The extension accepts `http` and `https` backend URLs. For local development this is reasonable, but for non-local use it means login credentials, tokens, titles, URLs, comments, timestamps, and tags can be transmitted without transport encryption.
Recommendation: enforce `https` for non-localhost backends, or at minimum warn before allowing a non-TLS backend.
### R2. Backend publication and retention are outside extension control
The extension's privacy behavior ends once data is sent to the configured LinkLog server. The backend may publish links publicly, combine them with user profile data, retain them, or forward them through enabled plugins such as Mastodon.
Recommendation: keep extension-facing privacy text explicit that saved link data becomes subject to the chosen backend's policies and configuration.
### R3. Persistent local account metadata is not cleared on sign-out
Sign-out clears session credentials, but leaves `backendUrl`, `email`, and `username` in persistent extension storage for convenience.
Recommendation: this is a reasonable default, but the UI could offer a separate "forget this account on this browser" action.
## Overall Conclusion
The Firefox extension has a relatively narrow privacy footprint and largely follows least-privilege principles. Its privacy disclosure about website activity is implemented and release-validated, session tokens are kept in session storage, and no unexpected telemetry or third-party exfiltration paths were found in the extension code.
The most important remaining privacy improvement is to prevent or strongly discourage non-HTTPS backends outside local development.
This document reflects the repository implementation reviewed on 2026-09-16. Reverse-proxy settings, browser privacy settings, database access, server logs, deployment networking, dependencies, and third-party Mastodon behavior can materially change the effective risk. A production deployment should supplement this review with configuration review, dependency and container scanning, authenticated dynamic tests, and a retention/access-control review.
+25 -9
View File
@@ -151,12 +151,6 @@ Run full backend and frontend test suites.
### Assistant outcome
The normal backend suite passed with 46 tests. Frontend and extension validation passed for all JavaScript syntax checks, JSON files, XPI rebuild, and unsigned XPI validation. The warnings-as-errors backend variant failed on unclosed SQLite connection `ResourceWarning` instances under Python 3.13; no functional tests failed.
### User
Validate the Privacy properties of the firefox plugin specifically and report them in Privacy.md
### Assistant outcome
Added `Privacy.md` with a plugin-specific privacy validation covering manifest permissions, active-tab capture scope, local and session storage, outbound network destinations, Firefox `data_collection_permissions`, release validation, and residual privacy risks. Confirmed the signed XPI release metadata validates successfully and identified non-HTTPS backend support as the main remaining privacy caveat.
### User
Address SA-007 and use linklog.example.com as default LINKLOG_PUBLIC_URL.
@@ -285,6 +279,12 @@ Add authenticated admin/user configuration
### Assistant outcome
Added shared bearer-token dependencies, protected user configuration endpoints, restricted plugin administration to administrator accounts, and changed user configuration to use the authenticated user's ID instead of hard-coded `user-1`. The seeded `alice` account is an administrator for development and `bob` remains a standard user. README authentication guidance was added, and the strict backend suite passed with 8 tests.
### User
What are the fingerprinting risks of this system?
### Assistant outcome
Reviewed the public feed, authentication and link APIs, frontend, Firefox extension, Mastodon integration, configuration, and existing security audit. Identified high privacy risk from public link and profile metadata, plus medium risks from deployment/configuration fingerprinting, extension website activity, external-request attribution, and observable authentication behavior. No code changes were made.
### User
Make sure the link-log page is responsive
@@ -1527,6 +1527,14 @@ In the mastodon post change the line: From:< URL>
to
Logged on <date> from: <url>
### User
I want to add functionality: a user should be able to send an email from the address associated with their account. When the email contains a link then that link will be stored in the LinkLog for that user. Also parse a potential comment and included tags.
The backend should be polling an IMAP account
### Assistant outcome
Added inbound email ingestion backed by IMAP polling. LinkLog now matches the sender against the user's verified primary or additional email addresses, extracts the first link plus comment and hashtags from the message, stores the link, and records processed message UIDs to avoid duplicates. Added encrypted IMAP settings storage, administrator IMAP configuration endpoints, startup polling integration, and regression coverage for parsing, sender matching, polling, and configuration checks.
Where date format is like: 2026 August 29 - 21:10
### Assistant outcome
@@ -1622,10 +1630,18 @@ Implement the same functionality in the firefox plugin
### Assistant outcome
Added the same live character-count warning to the Firefox extension popup: it fetches `/api/public/config` for the limit and the user's `/api/user/plugins/mastodon` post prefix, estimates the assembled Mastodon post length the same way as the New Entry page and backend plugin, and shows a localized `used/limit characters` counter (with an over-limit warning message) that updates on every relevant input and tag toggle. Added the `characterCount`/`characterCountOverLimit` translations to all five locale files (en-US, de, es, fr, nl) and verified all locale JSON files parse and `popup.js` passes a Node syntax check.
# Chat Log
### User
Make those colors less outstanding, this should not stand out. In fact, give the box a title :profile and make it collapsable
Add these findings and recomendations to Privacy.md
### Assistant outcome
Made the feed profile summary use the normal subdued link-item colors and converted it to a native collapsible `<details>` section titled `:profile`. Template and stylesheet diagnostics passed.
# Chat Log
Created `Privacy.md` with a source-grounded privacy and fingerprinting assessment covering public activity correlation, URL and query-parameter leakage, Firefox extension website activity, Mastodon correlation, server-side URL fetching, deployment fingerprinting, authentication observability, user guidance, and prioritized mitigations. Validated the document contents; no runtime code was changed.
### User
I want to add functionality: a user should be able to send an email from the address associated with their account. When the email contains a link then that link will be stored in the LinkLog for that user. Also parse a potential comment and included tags.
The backend should be polling an IMAP account
### Assistant outcome
Added inbound email ingestion backed by IMAP polling. LinkLog now matches the sender against the user's verified primary or additional email addresses, extracts the first link plus comment and hashtags from the message, stores the link, and records processed message UIDs to avoid duplicates. Added encrypted IMAP settings storage, administrator IMAP configuration endpoints, startup polling integration, and regression coverage for parsing, sender matching, polling, and configuration checks.
+8 -10
View File
@@ -278,6 +278,7 @@
248. Release script fails because the runner's hashlib module has no file_digest attribute.
249. The filtering on tags does not work. When I click on the button I cannot select. Remember that any visitor on the site should be able to filter the tag for the link-feed.
250. Update VIBE and Changelog
251. What are the fingerprinting risks of this system?
## 2026-08-30
@@ -298,10 +299,6 @@ Where date format is like: 2026 August 29 - 21:10
## 2026-09-05
254. In firefox the search input field is wider (at 180px) then labe search control (at 135.5px)
## 2026-09-07
255. Change bg color and slightly decrease the font for 'html body main.container section.link-item.profile-summary' so that it stands out
255. In the toolbar switch the search and tag filter's location
## 2026-09-06
@@ -318,15 +315,16 @@ If the user uses the filters and/or search in the toolbox then those should limi
259. Make sure the CHANGELOG reflects all changes since commit 74b2c400c6
260. Add a skill to maintain the changelog.md
261. In the new-entry form, provide a warning when the total ammount of characters is over LINKLOG_MAX_POST_CHARACTERS=500 (also indicat the number of characters used/500 count). This is to prevent Mastodon posts from failing - so all characters should be counted.
## 2026-09-15
262. Validate the Privacy properties of the firefox plugin specifically and report them in Privacy.md
262. Implement the same functionality in the firefox plugin
## 2026-09-07
## 2026-09-16
Make those colors less outstanding, this should not stand out. In fact, give the box a title :profile and make it collapsable
263. Add these findings and recomendations to Privacy.md
## 2026-09-30
1. I want to add functionality: a user should be able to send an email from the address associated with their account. When the email contains a link then that link will be stored in the LinkLog for that user. Also parse a potential comment and included tags.
2. The backend should be polling an IMAP account
## Future entries
Binary file not shown.
+66
View File
@@ -18,6 +18,7 @@ from backend.app.services.email_service import (
send_verification_email,
smtp_configured,
)
from backend.app.services.inbound_email_service import get_imap_settings, imap_configured, save_imap_settings
from backend.app.services.email_verification import create_verification_token
from backend.app.services.theme_service import THEMES, get_enabled_themes, save_enabled_themes
from backend.app.services.secret_store import encrypt_secret
@@ -63,6 +64,16 @@ class AdminThemesUpdate(BaseModel):
themes: list[str]
class AdminImapUpdate(BaseModel):
imap_host: str
imap_port: int = 993
imap_username: str = ''
imap_password: str = ''
imap_mailbox: str = 'INBOX'
imap_use_ssl: bool = True
imap_poll_interval_seconds: int = 60
def validate_smtp_values(payload: AdminSmtpUpdate, current: dict | None = None) -> dict:
smtp_host = payload.smtp_host.strip()
smtp_from = payload.smtp_from.strip()
@@ -80,6 +91,27 @@ def validate_smtp_values(payload: AdminSmtpUpdate, current: dict | None = None)
}
def validate_imap_values(payload: AdminImapUpdate, current: dict | None = None) -> dict:
imap_host = payload.imap_host.strip()
imap_username = payload.imap_username.strip()
imap_mailbox = payload.imap_mailbox.strip() or 'INBOX'
if not imap_host or not imap_username or not imap_mailbox:
raise HTTPException(status_code=422, detail='IMAP host, username, and mailbox are required')
if not 1 <= payload.imap_port <= 65535:
raise HTTPException(status_code=422, detail='IMAP port must be between 1 and 65535')
if payload.imap_poll_interval_seconds < 5:
raise HTTPException(status_code=422, detail='IMAP poll interval must be at least 5 seconds')
return {
'imap_host': imap_host,
'imap_port': payload.imap_port,
'imap_username': imap_username,
'imap_password': payload.imap_password or (current or {}).get('imap_password', ''),
'imap_mailbox': imap_mailbox,
'imap_use_ssl': payload.imap_use_ssl,
'imap_poll_interval_seconds': payload.imap_poll_interval_seconds,
}
def public_user(row):
return {
'id': row['id'],
@@ -166,11 +198,29 @@ def public_smtp_settings(values: dict) -> dict:
}
def public_imap_settings(values: dict) -> dict:
return {
'imap_host': values['imap_host'],
'imap_port': values['imap_port'],
'imap_username': values['imap_username'],
'imap_mailbox': values['imap_mailbox'],
'imap_use_ssl': values['imap_use_ssl'],
'imap_poll_interval_seconds': values['imap_poll_interval_seconds'],
'password_configured': bool(values['imap_password']),
'configured': imap_configured(values),
}
@router.get('/smtp')
def get_admin_smtp_settings(_: dict = Depends(require_admin)):
return public_smtp_settings(get_smtp_settings())
@router.get('/imap')
def get_admin_imap_settings(_: dict = Depends(require_admin)):
return public_imap_settings(get_imap_settings())
@router.get('/themes')
def get_admin_themes(_: dict = Depends(require_admin)):
return {'themes': THEMES, 'enabled': get_enabled_themes()}
@@ -195,6 +245,22 @@ def update_admin_smtp_settings(payload: AdminSmtpUpdate, current_user: dict = De
return public_smtp_settings(values)
@router.put('/imap')
def update_admin_imap_settings(payload: AdminImapUpdate, current_user: dict = Depends(require_admin)):
current = get_imap_settings()
values = validate_imap_values(payload, current)
save_imap_settings(values)
record_audit_event(current_user['id'], 'imap_settings_updated', 'application', details={
'host': values['imap_host'],
'port': values['imap_port'],
'username': values['imap_username'],
'mailbox': values['imap_mailbox'],
'ssl': values['imap_use_ssl'],
'poll_interval_seconds': values['imap_poll_interval_seconds'],
})
return public_imap_settings(values)
@router.post('/smtp/test')
def validate_admin_smtp(payload: AdminSmtpUpdate, request: Request, current_user: dict = Depends(require_admin)):
values = validate_smtp_values(payload, get_smtp_settings())
+7
View File
@@ -47,6 +47,13 @@ class Settings:
smtp_password: str = os.getenv('LINKLOG_SMTP_PASSWORD', '')
smtp_from: str = os.getenv('LINKLOG_SMTP_FROM', 'LinkLog <no-reply@localhost>')
smtp_use_tls: bool = os.getenv('LINKLOG_SMTP_USE_TLS', 'true').lower() in {'1', 'true', 'yes'}
imap_host: str = os.getenv('LINKLOG_IMAP_HOST', '')
imap_port: int = int(os.getenv('LINKLOG_IMAP_PORT', '993'))
imap_username: str = os.getenv('LINKLOG_IMAP_USERNAME', '')
imap_password: str = os.getenv('LINKLOG_IMAP_PASSWORD', '')
imap_mailbox: str = os.getenv('LINKLOG_IMAP_MAILBOX', 'INBOX')
imap_use_ssl: bool = os.getenv('LINKLOG_IMAP_USE_SSL', 'true').lower() in {'1', 'true', 'yes'}
imap_poll_interval_seconds: int = int(os.getenv('LINKLOG_IMAP_POLL_INTERVAL_SECONDS', '60'))
email_verification_expiry_hours: int = int(os.getenv('LINKLOG_EMAIL_VERIFICATION_EXPIRY_HOURS', '24'))
password_reset_expiry_hours: int = int(os.getenv('LINKLOG_PASSWORD_RESET_EXPIRY_HOURS', '1'))
mastodon_client_name: str = os.getenv('LINKLOG_MASTODON_CLIENT_NAME', 'LinkLog')
+16
View File
@@ -259,6 +259,22 @@ CREATE TABLE IF NOT EXISTS security_audit_events (
);
CREATE INDEX IF NOT EXISTS idx_security_audit_events_created_at ON security_audit_events(created_at);
CREATE INDEX IF NOT EXISTS idx_security_audit_events_actor_id ON security_audit_events(actor_id);
'''),
(18, '''
CREATE TABLE IF NOT EXISTS inbound_email_messages (
mailbox TEXT NOT NULL,
uid TEXT NOT NULL,
message_id TEXT,
user_id TEXT,
link_id TEXT,
status TEXT NOT NULL,
details TEXT NOT NULL DEFAULT '{}',
created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (mailbox, uid),
FOREIGN KEY(user_id) REFERENCES users(id) ON DELETE SET NULL,
FOREIGN KEY(link_id) REFERENCES links(id) ON DELETE SET NULL
);
CREATE INDEX IF NOT EXISTS idx_inbound_email_messages_message_id ON inbound_email_messages(message_id);
''')
]
+22 -1
View File
@@ -1,8 +1,10 @@
## Copyright © 2026 Olaf Kolkman
## SPDX-License-Identifier: GPL-3.0-or-later
from contextlib import asynccontextmanager
from fastapi import FastAPI, Request
import logging
from threading import Event, Thread
from uuid import uuid4
from fastapi.responses import HTMLResponse
from fastapi.responses import RedirectResponse
@@ -20,12 +22,31 @@ from backend.app.api.setup import has_administrator
from backend.app.api.user_config import router as user_config_router
from backend.app.core.config import settings, validate_configuration
from backend.app.database import AVATARS_DIR
from backend.app.services.inbound_email_service import run_imap_polling
from backend.app.services.link_service import get_public_profile
logging.basicConfig(level=getattr(logging, settings.log_level, logging.INFO))
validate_configuration(settings)
app = FastAPI(title='LinkLog API', version=settings.version)
@asynccontextmanager
async def lifespan(_: FastAPI):
stop_event = Event()
worker = Thread(
target=run_imap_polling,
args=(stop_event, logging.getLogger('backend.app.services.inbound_email_service')),
name='linklog-imap-poller',
daemon=True,
)
worker.start()
try:
yield
finally:
stop_event.set()
worker.join(timeout=2)
app = FastAPI(title='LinkLog API', version=settings.version, lifespan=lifespan)
@app.middleware('http')
@@ -0,0 +1,308 @@
## Copyright © 2026 Olaf Kolkman
## SPDX-License-Identifier: GPL-3.0-or-later
from email import policy
from email.header import decode_header, make_header
from email.parser import BytesParser
from email.utils import getaddresses
from html.parser import HTMLParser
import imaplib
import json
import logging
import re
from threading import Event
from backend.app.core.config import settings
from backend.app.database import get_connection
from backend.app.services.link_service import create_link, normalize_tags
from backend.app.services.secret_store import decrypt_secret, encrypt_secret
URL_PATTERN = re.compile(r'https?://[^\s<>()"\']+')
HASHTAG_PATTERN = re.compile(r'(?<!\w)#([A-Za-z0-9][\w-]*)')
class _HTMLTextExtractor(HTMLParser):
def __init__(self) -> None:
super().__init__()
self.text_parts: list[str] = []
self.hrefs: list[str] = []
def handle_data(self, data: str) -> None:
if data:
self.text_parts.append(data)
def handle_starttag(self, tag: str, attrs: list[tuple[str, str | None]]) -> None:
if tag.lower() != 'a':
return
for name, value in attrs:
if name.lower() == 'href' and value:
self.hrefs.append(value)
def _decode_header_value(value: str | None) -> str:
if not value:
return ''
return str(make_header(decode_header(value))).strip()
def _extract_message_text(message) -> tuple[str, list[str]]:
plain_parts: list[str] = []
html_parts: list[str] = []
html_hrefs: list[str] = []
parts = message.walk() if message.is_multipart() else [message]
for part in parts:
if part.get_content_maintype() == 'multipart':
continue
content_type = part.get_content_type()
try:
content = part.get_content()
except (LookupError, UnicodeDecodeError):
continue
if not isinstance(content, str):
continue
if content_type == 'text/plain':
plain_parts.append(content)
elif content_type == 'text/html':
parser = _HTMLTextExtractor()
parser.feed(content)
html_parts.append(' '.join(parser.text_parts))
html_hrefs.extend(parser.hrefs)
text = '\n\n'.join(part.strip() for part in (plain_parts or html_parts) if part.strip())
urls = list(dict.fromkeys([*html_hrefs, *URL_PATTERN.findall(text)]))
return text, urls
def _extract_comment(text: str) -> str:
without_links = URL_PATTERN.sub(' ', text)
without_tags = HASHTAG_PATTERN.sub(' ', without_links)
lines = []
for raw_line in without_tags.splitlines():
line = ' '.join(raw_line.split()).strip()
if line:
lines.append(line)
return '\n'.join(lines)
def _extract_tags(*values: str) -> list[str]:
return normalize_tags([match.group(0) for value in values for match in HASHTAG_PATTERN.finditer(value or '')])
def _resolve_sender_email(message) -> str:
addresses = getaddresses(message.get_all('from', []))
for _, email in addresses:
if email:
return email.strip().lower()
return ''
def parse_incoming_email(raw_message: bytes) -> dict:
message = BytesParser(policy=policy.default).parsebytes(raw_message)
subject = _decode_header_value(message.get('Subject'))
text, urls = _extract_message_text(message)
subject_without_tags = HASHTAG_PATTERN.sub(' ', subject)
subject_without_link = URL_PATTERN.sub(' ', subject_without_tags)
title = ' '.join(subject_without_link.split()).strip()
tags = _extract_tags(subject, text)
return {
'message_id': _decode_header_value(message.get('Message-ID')),
'from_email': _resolve_sender_email(message),
'subject': subject,
'title': title,
'comment': _extract_comment(text),
'tags': tags,
'url': urls[0] if urls else None,
}
def get_imap_settings() -> dict:
values = {
'imap_host': settings.imap_host,
'imap_port': settings.imap_port,
'imap_username': settings.imap_username,
'imap_password': settings.imap_password,
'imap_mailbox': settings.imap_mailbox,
'imap_use_ssl': settings.imap_use_ssl,
'imap_poll_interval_seconds': settings.imap_poll_interval_seconds,
}
with get_connection() as conn:
row = conn.execute('SELECT value FROM app_settings WHERE name = ?', ('imap',)).fetchone()
if row:
values.update(json.loads(row['value']))
values['imap_password'] = decrypt_secret(values['imap_password'])
return values
def save_imap_settings(values: dict) -> None:
stored_values = {
**values,
'imap_password': encrypt_secret(values['imap_password']),
}
with get_connection() as conn:
conn.execute(
'''INSERT INTO app_settings (name, value, updated_at) VALUES (?, ?, CURRENT_TIMESTAMP)
ON CONFLICT(name) DO UPDATE SET value = excluded.value, updated_at = CURRENT_TIMESTAMP''',
('imap', json.dumps(stored_values)),
)
conn.commit()
def imap_configured(imap_values: dict | None = None) -> bool:
imap = imap_values or get_imap_settings()
return bool(imap['imap_host'] and imap['imap_username'] and imap['imap_password'] and imap['imap_mailbox'])
def _find_user_by_email(address: str) -> dict | None:
email = address.strip().lower()
if not email:
return None
with get_connection() as conn:
row = conn.execute(
'SELECT id, username FROM users WHERE lower(email) = ? AND email_verified = 1',
(email,),
).fetchone()
if row is None:
row = conn.execute(
'''SELECT users.id, users.username
FROM user_email_addresses
JOIN users ON users.id = user_email_addresses.user_id
WHERE lower(user_email_addresses.email) = ? AND user_email_addresses.verified = 1''',
(email,),
).fetchone()
return dict(row) if row else None
def _already_processed(mailbox: str, uid: str) -> bool:
with get_connection() as conn:
row = conn.execute(
'SELECT 1 FROM inbound_email_messages WHERE mailbox = ? AND uid = ? LIMIT 1',
(mailbox, uid),
).fetchone()
return row is not None
def _record_message(mailbox: str, uid: str, status: str, parsed: dict, user_id: str | None = None, link_id: str | None = None, details: dict | None = None) -> None:
payload = details or {}
with get_connection() as conn:
conn.execute(
'''INSERT INTO inbound_email_messages (mailbox, uid, message_id, user_id, link_id, status, details)
VALUES (?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(mailbox, uid) DO UPDATE SET
message_id = excluded.message_id,
user_id = excluded.user_id,
link_id = excluded.link_id,
status = excluded.status,
details = excluded.details''',
(mailbox, uid, parsed.get('message_id'), user_id, link_id, status, json.dumps(payload)),
)
conn.commit()
def ingest_message(mailbox: str, uid: str, raw_message: bytes) -> dict:
if _already_processed(mailbox, uid):
return {'status': 'duplicate', 'mailbox': mailbox, 'uid': uid}
parsed = parse_incoming_email(raw_message)
sender = parsed['from_email']
if not sender:
_record_message(mailbox, uid, 'ignored_missing_sender', parsed)
return {'status': 'ignored_missing_sender', 'mailbox': mailbox, 'uid': uid}
user = _find_user_by_email(sender)
if user is None:
_record_message(mailbox, uid, 'ignored_unknown_sender', parsed, details={'from_email': sender})
return {'status': 'ignored_unknown_sender', 'mailbox': mailbox, 'uid': uid, 'from_email': sender}
if not parsed['url']:
_record_message(mailbox, uid, 'ignored_no_link', parsed, user_id=user['id'])
return {'status': 'ignored_no_link', 'mailbox': mailbox, 'uid': uid, 'user_id': user['id']}
title = parsed['title'] or parsed['url']
record = create_link(user['id'], title, parsed['url'], parsed['comment'], None, parsed['tags'])
_record_message(mailbox, uid, 'stored', parsed, user_id=user['id'], link_id=record['id'])
return {
'status': 'stored',
'mailbox': mailbox,
'uid': uid,
'user_id': user['id'],
'link_id': record['id'],
'link': record,
}
def _open_imap_client(imap_values: dict):
client_class = imaplib.IMAP4_SSL if imap_values['imap_use_ssl'] else imaplib.IMAP4
return client_class(imap_values['imap_host'], imap_values['imap_port'])
def _fetch_message_bytes(response_data) -> bytes | None:
for item in response_data or []:
if isinstance(item, tuple) and len(item) > 1 and isinstance(item[1], (bytes, bytearray)):
return bytes(item[1])
return None
def poll_inbox_once(imap_values: dict | None = None, client_factory=None) -> dict:
values = imap_values or get_imap_settings()
if not imap_configured(values):
return {'status': 'not_configured', 'processed': 0, 'stored': 0, 'ignored': 0}
mailbox = values['imap_mailbox']
client = (client_factory or _open_imap_client)(values)
processed = 0
stored = 0
ignored = 0
try:
client.login(values['imap_username'], values['imap_password'])
status, _ = client.select(mailbox)
if status != 'OK':
raise RuntimeError(f'Could not select IMAP mailbox {mailbox}')
status, data = client.uid('search', None, 'UNSEEN')
if status != 'OK':
raise RuntimeError('Could not list unseen IMAP messages')
raw_uids = data[0].split() if data and data[0] else []
for uid in raw_uids:
uid_value = uid.decode('utf-8') if isinstance(uid, bytes) else str(uid)
status, message_data = client.uid('fetch', uid, '(BODY.PEEK[])')
if status != 'OK':
raise RuntimeError(f'Could not fetch IMAP message {uid_value}')
raw_message = _fetch_message_bytes(message_data)
if raw_message is None:
continue
result = ingest_message(mailbox, uid_value, raw_message)
processed += 1
if result['status'] == 'stored':
stored += 1
else:
ignored += 1
if result['status'] in {
'stored', 'duplicate', 'ignored_missing_sender', 'ignored_unknown_sender', 'ignored_no_link',
}:
client.uid('store', uid, '+FLAGS', '(\\Seen)')
return {'status': 'ok', 'processed': processed, 'stored': stored, 'ignored': ignored}
finally:
try:
client.logout()
except Exception:
pass
def run_imap_polling(stop_event: Event, logger: logging.Logger | None = None) -> None:
active_logger = logger or logging.getLogger(__name__)
while not stop_event.is_set():
values = get_imap_settings()
interval = max(5, int(values.get('imap_poll_interval_seconds', settings.imap_poll_interval_seconds or 60)))
if not imap_configured(values):
stop_event.wait(interval)
continue
try:
summary = poll_inbox_once(values)
if summary['processed']:
active_logger.info(
'IMAP poll processed=%s stored=%s ignored=%s mailbox=%s',
summary['processed'], summary['stored'], summary['ignored'], values['imap_mailbox'],
)
except Exception as error:
active_logger.warning('IMAP polling failed mailbox=%s error=%s', values.get('imap_mailbox'), error)
stop_event.wait(interval)
+102
View File
@@ -15,6 +15,7 @@ from backend.app.main import app
from backend.app.core.config import settings
from backend.app.database import get_connection, hash_password
from backend.app.services.email_service import get_smtp_settings
from backend.app.services.inbound_email_service import get_imap_settings, poll_inbox_once
from backend.app.services.login_throttle import clear_login_failures
from backend.app.services.otp_service import current_code
from backend.app.services.password_reset import create_reset_token
@@ -147,6 +148,8 @@ def test_configuration_requires_authentication_and_admin_role():
assert client.get('/api/admin/users', headers=login_headers('bob')).status_code == 403
assert client.get('/api/admin/smtp').status_code == 401
assert client.get('/api/admin/smtp', headers=login_headers('bob')).status_code == 403
assert client.get('/api/admin/imap').status_code == 401
assert client.get('/api/admin/imap', headers=login_headers('bob')).status_code == 403
def test_admin_can_select_multiple_themes():
@@ -234,6 +237,105 @@ def test_admin_reports_smtp_validation_errors():
assert failed_validation.headers['X-Request-ID']
def test_admin_can_save_imap_settings():
headers = login_headers()
original = get_imap_settings()
with get_connection() as conn:
original_row = conn.execute('SELECT value FROM app_settings WHERE name = ?', ('imap',)).fetchone()
response = client.put('/api/admin/imap', headers=headers, json={
'imap_host': 'imap.example.com',
'imap_port': 993,
'imap_username': 'collector@example.com',
'imap_password': 'secret-imap-password',
'imap_mailbox': 'INBOX',
'imap_use_ssl': True,
'imap_poll_interval_seconds': 60,
})
assert response.status_code == 200
assert response.json()['imap_host'] == 'imap.example.com'
assert response.json()['password_configured'] is True
assert response.json()['configured'] is True
assert 'imap_password' not in response.json()
if original_row is None:
with get_connection() as conn:
conn.execute('DELETE FROM app_settings WHERE name = ?', ('imap',))
conn.commit()
else:
with get_connection() as conn:
conn.execute(
'UPDATE app_settings SET value = ?, updated_at = CURRENT_TIMESTAMP WHERE name = ?',
(original_row['value'], 'imap'),
)
conn.commit()
def test_poll_inbox_once_reads_unseen_messages_and_marks_them_seen():
class FakeImapClient:
def __init__(self):
self.actions = []
self.raw_message = (
b'From: Alice <alice@example.com>\n'
b'To: capture@linklog.example\n'
b'Subject: Polled inbox entry #AI\n'
b'Message-ID: <poll-1@example.com>\n'
b'Content-Type: text/plain; charset="utf-8"\n\n'
b'Collected from IMAP.\nhttps://example.com/polled\n'
)
def login(self, username, password):
self.actions.append(('login', username, password))
return 'OK', [b'Logged in']
def select(self, mailbox):
self.actions.append(('select', mailbox))
return 'OK', [b'1']
def uid(self, command, *args):
self.actions.append(('uid', command, *args))
if command == 'search':
return 'OK', [b'200']
if command == 'fetch':
return 'OK', [(b'200 (BODY[] {42}', self.raw_message), b')']
if command == 'store':
return 'OK', [b'200 (FLAGS (\\Seen))']
raise AssertionError(command)
def logout(self):
self.actions.append(('logout',))
return 'BYE', [b'Logged out']
summary = poll_inbox_once(
{
'imap_host': 'imap.example.com',
'imap_port': 993,
'imap_username': 'collector@example.com',
'imap_password': 'secret',
'imap_mailbox': 'INBOX',
'imap_use_ssl': True,
'imap_poll_interval_seconds': 60,
},
client_factory=lambda _: FakeImapClient(),
)
assert summary == {'status': 'ok', 'processed': 1, 'stored': 1, 'ignored': 0}
with get_connection() as conn:
link = conn.execute(
'SELECT title, url, comment, user_id FROM links WHERE url = ?',
('https://example.com/polled',),
).fetchone()
inbound = conn.execute(
'SELECT status FROM inbound_email_messages WHERE mailbox = ? AND uid = ?',
('INBOX', '200'),
).fetchone()
assert link['user_id'] == 'user-1'
assert link['title'] == 'Polled inbox entry'
assert link['comment'] == 'Collected from IMAP.'
assert inbound['status'] == 'stored'
def test_request_id_is_preserved_and_sensitive_error_text_is_not_returned():
headers = login_headers()
with get_connection() as conn:
@@ -0,0 +1,92 @@
## Copyright © 2026 Olaf Kolkman
## SPDX-License-Identifier: GPL-3.0-or-later
from email.message import EmailMessage
from backend.app.database import get_connection
from backend.app.services.inbound_email_service import ingest_message, parse_incoming_email
def _raw_message(sender: str, subject: str, body: str, message_id: str) -> bytes:
message = EmailMessage()
message['From'] = sender
message['To'] = 'capture@linklog.example'
message['Subject'] = subject
message['Message-ID'] = message_id
message.set_content(body)
return message.as_bytes()
def test_parse_incoming_email_extracts_title_comment_tags_and_link():
parsed = parse_incoming_email(_raw_message(
'Alice <alice@example.com>',
'A useful article #AI',
'Interesting summary about the protocol.\nhttps://example.com/posts/1\n#Security',
'<msg-1@example.com>',
))
assert parsed['from_email'] == 'alice@example.com'
assert parsed['title'] == 'A useful article'
assert parsed['url'] == 'https://example.com/posts/1'
assert parsed['comment'] == 'Interesting summary about the protocol.'
assert parsed['tags'] == ['#AI', '#Security']
def test_ingest_message_stores_link_for_primary_email_sender():
result = ingest_message(
'INBOX',
'101',
_raw_message(
'Alice <alice@example.com>',
'Email sourced link #Fediverse',
'A comment from the inbox.\nhttps://example.com/inbox-link',
'<msg-2@example.com>',
),
)
assert result['status'] == 'stored'
with get_connection() as conn:
link = conn.execute(
'SELECT title, url, comment, user_id FROM links WHERE id = ?',
(result['link_id'],),
).fetchone()
tags = conn.execute(
'''SELECT tags.name
FROM tags
JOIN link_tags ON link_tags.tag_id = tags.id
WHERE link_tags.link_id = ?
ORDER BY tags.name''',
(result['link_id'],),
).fetchall()
audit = conn.execute(
'SELECT status FROM inbound_email_messages WHERE mailbox = ? AND uid = ?',
('INBOX', '101'),
).fetchone()
assert link['user_id'] == 'user-1'
assert link['title'] == 'Email sourced link'
assert link['url'] == 'https://example.com/inbox-link'
assert link['comment'] == 'A comment from the inbox.'
assert [row['name'] for row in tags] == ['#Fediverse']
assert audit['status'] == 'stored'
def test_ingest_message_accepts_verified_additional_address_and_ignores_duplicates():
with get_connection() as conn:
conn.execute(
'INSERT INTO user_email_addresses (id, user_id, email, verified) VALUES (?, ?, ?, 1)',
('alt-email-test', 'user-1', 'alice-alt@example.com'),
)
conn.commit()
raw_message = _raw_message(
'Alice Alt <alice-alt@example.com>',
'Secondary sender #Links',
'https://example.com/secondary',
'<msg-3@example.com>',
)
first = ingest_message('INBOX', '102', raw_message)
second = ingest_message('INBOX', '102', raw_message)
assert first['status'] == 'stored'
assert second['status'] == 'duplicate'
+7
View File
@@ -20,6 +20,13 @@ services:
LINKLOG_SMTP_PASSWORD: ${LINKLOG_SMTP_PASSWORD:?Set LINKLOG_SMTP_PASSWORD in .env}
LINKLOG_SMTP_FROM: ${LINKLOG_SMTP_FROM:-LinkLog <no-reply@example.com>}
LINKLOG_SMTP_USE_TLS: ${LINKLOG_SMTP_USE_TLS:-true}
LINKLOG_IMAP_HOST: ${LINKLOG_IMAP_HOST:-imap.example.com}
LINKLOG_IMAP_PORT: ${LINKLOG_IMAP_PORT:-993}
LINKLOG_IMAP_USERNAME: ${LINKLOG_IMAP_USERNAME:?Set LINKLOG_IMAP_USERNAME in .env}
LINKLOG_IMAP_PASSWORD: ${LINKLOG_IMAP_PASSWORD:?Set LINKLOG_IMAP_PASSWORD in .env}
LINKLOG_IMAP_MAILBOX: ${LINKLOG_IMAP_MAILBOX:-INBOX}
LINKLOG_IMAP_USE_SSL: ${LINKLOG_IMAP_USE_SSL:-true}
LINKLOG_IMAP_POLL_INTERVAL_SECONDS: ${LINKLOG_IMAP_POLL_INTERVAL_SECONDS:-60}
LINKLOG_EMAIL_VERIFICATION_EXPIRY_HOURS: ${LINKLOG_EMAIL_VERIFICATION_EXPIRY_HOURS:-24}
LINKLOG_PASSWORD_RESET_EXPIRY_HOURS: ${LINKLOG_PASSWORD_RESET_EXPIRY_HOURS:-1}
LINKLOG_MASTODON_CLIENT_NAME: ${LINKLOG_MASTODON_CLIENT_NAME:-LinkLog}
+7
View File
@@ -19,6 +19,13 @@ services:
LINKLOG_PUBLIC_URL: ${LINKLOG_PUBLIC_URL:-http://localhost:8000}
LINKLOG_TOKEN_EXPIRY_MINUTES: ${LINKLOG_TOKEN_EXPIRY_MINUTES:-15}
LINKLOG_REFRESH_TOKEN_EXPIRY_DAYS: ${LINKLOG_REFRESH_TOKEN_EXPIRY_DAYS:-30}
LINKLOG_IMAP_HOST: ${LINKLOG_IMAP_HOST:-}
LINKLOG_IMAP_PORT: ${LINKLOG_IMAP_PORT:-993}
LINKLOG_IMAP_USERNAME: ${LINKLOG_IMAP_USERNAME:-}
LINKLOG_IMAP_PASSWORD: ${LINKLOG_IMAP_PASSWORD:-}
LINKLOG_IMAP_MAILBOX: ${LINKLOG_IMAP_MAILBOX:-INBOX}
LINKLOG_IMAP_USE_SSL: ${LINKLOG_IMAP_USE_SSL:-true}
LINKLOG_IMAP_POLL_INTERVAL_SECONDS: ${LINKLOG_IMAP_POLL_INTERVAL_SECONDS:-60}
LINKLOG_LOG_LEVEL: ${LINKLOG_LOG_LEVEL:-DEBUG}
LINKLOG_TRACKING_PARAMS: ${LINKLOG_TRACKING_PARAMS:-}
restart: ${APP_RESTART_POLICY:-unless-stopped}
+7
View File
@@ -19,6 +19,13 @@ services:
LINKLOG_LOG_LEVEL: ${LINKLOG_LOG_LEVEL:-INFO}
LINKLOG_TOKEN_EXPIRY_MINUTES: ${LINKLOG_TOKEN_EXPIRY_MINUTES:-15}
LINKLOG_REFRESH_TOKEN_EXPIRY_DAYS: ${LINKLOG_REFRESH_TOKEN_EXPIRY_DAYS:-30}
LINKLOG_IMAP_HOST: ${LINKLOG_IMAP_HOST:-}
LINKLOG_IMAP_PORT: ${LINKLOG_IMAP_PORT:-993}
LINKLOG_IMAP_USERNAME: ${LINKLOG_IMAP_USERNAME:-}
LINKLOG_IMAP_PASSWORD: ${LINKLOG_IMAP_PASSWORD:-}
LINKLOG_IMAP_MAILBOX: ${LINKLOG_IMAP_MAILBOX:-INBOX}
LINKLOG_IMAP_USE_SSL: ${LINKLOG_IMAP_USE_SSL:-true}
LINKLOG_IMAP_POLL_INTERVAL_SECONDS: ${LINKLOG_IMAP_POLL_INTERVAL_SECONDS:-60}
LINKLOG_TRACKING_PARAMS: ${LINKLOG_TRACKING_PARAMS:-}
restart: ${APP_RESTART_POLICY:-unless-stopped}
healthcheck: