Merge branch 'wukko:current' into ddinstagram

README.md

@@ -4,7 +4,7 @@ best way to save what you love: [cobalt.tools](https://cobalt.tools/)
 ![cobalt logo with repeated logo (double arrow) pattern background](https://raw.githubusercontent.com/wukko/cobalt/current/src/front/icons/pattern.png "cobalt logo with repeated logo (double arrow) pattern background")  
 
 ## what's cobalt?
-cobalt is a media downloader that doesn't piss you off. it's fast, friendly, and doesn't have any bullshit that modern web is filled with: ***no ads, trackers, or analytics***.  
+cobalt is a media downloader that doesn't piss you off. it's fast, friendly, and doesn't have any bullshit that modern web is filled with: ***no ads, trackers, or invasive analytics***.  
 
 paste the link, get the file, move on. it's that simple. just how it should be.
 
@@ -67,8 +67,6 @@ cobalt is a tool for easing content downloads from internet and takes ***zero li
 
 cobalt is ***NOT*** a piracy tool and cannot be used as such. it can only download free, publicly accessible content. such content can be easily downloaded through any browser's dev tools. pressing one button is easier, so i made a convenient, ad-less tool for such repeated actions.
 
-cobalt is my passion project, update schedule depends solely on my free time, motivation, and mood. don't expect any consistency in update releases.
-
 ## cobalt license
 cobalt code is licensed under [AGPL-3.0](https://github.com/wukko/cobalt/blob/current/LICENSE).
 
@@ -121,7 +119,7 @@ cobalt also depends on:
 - [node-cache](https://www.npmjs.com/package/node-cache) to cache stream info in server ram for a limited amount of time.
 - [psl](https://www.npmjs.com/package/psl) as the domain name parser.
 - [set-cookie-parser](https://www.npmjs.com/package/set-cookie-parser) to parse cookies that cobalt receives from certain services.
-- [undici](https://www.npmjs.com/package/undici) for making http requests
+- [undici](https://www.npmjs.com/package/undici) for making http requests.
 - [url-pattern](https://www.npmjs.com/package/url-pattern) to match provided links with supported patterns.
 
 ...and many other packages that these packages rely on.

docs/examples/docker-compose.example.yml

@@ -8,7 +8,7 @@ services:
 
         init: true
 
-        # if container doesn't run detached on your machine, uncomment the next line:
+        # if container doesn't run detached on your machine, uncomment the next line
         #tty: true
 
         ports:
@@ -18,12 +18,10 @@ services:
 
         environment:
             # replace https://co.wuk.sh/ with your instance's target url in same format
-            - API_URL=https://co.wuk.sh/
+            API_URL: "https://co.wuk.sh/"
             # replace eu-nl with your instance's distinctive name
-            - API_NAME=eu-nl
+            API_NAME: "eu-nl"
-            # if you want to use cookies when fetching data from services, uncomment the next line
+            # see docs/run-an-instance.md for more information
-            #- COOKIE_PATH=/cookies.json
-            # see cookies.example.json for example file.
         labels:
             - com.centurylinklabs.watchtower.scope=cobalt
 
@@ -38,7 +36,7 @@ services:
 
         init: true
 
-        # if container doesn't run detached on your machine, uncomment the next line:
+        # if container doesn't run detached on your machine, uncomment the next line
         #tty: true
 
         ports:
@@ -48,9 +46,9 @@ services:
 
         environment:
             # replace https://cobalt.tools/ with your instance's target url in same format
-            - WEB_URL=https://cobalt.tools/
+            WEB_URL: "https://cobalt.tools/"
             # replace https://co.wuk.sh/ with preferred api instance url
-            - API_URL=https://co.wuk.sh/
+            API_URL: "https://co.wuk.sh/"
 
         labels:
             - com.centurylinklabs.watchtower.scope=cobalt

docs/run-an-instance.md

@@ -59,9 +59,32 @@ sudo service nscd start
 | `CORS_URL`            | not used  | `https://cobalt.tools/` | cross-origin resource sharing url. api will be available only from this url if `CORS_WILDCARD` is set to `0`. |
 | `COOKIE_PATH`         | not used  | `/cookies.json`         | path for cookie file relative to main folder. |
 | `PROCESSING_PRIORITY` | not used  | `10`                    | changes `nice` value* for ffmpeg subprocess. available only on unix systems. |
+| `TIKTOK_DEVICE_INFO`  | ➖        | *see below*                    | device info (including `iid` and `device_id`) for tiktok functionality. required for tiktok to work. |
 
 \* the higher the nice value, the lower the priority. [read more here](https://en.wikipedia.org/wiki/Nice_(Unix)).
 
+#### TIKTOK_DEVICE_INFO
+you need to get your own device info for tiktok functionality to work. this can be done by proxying the app through any request-intercepting proxy (such as [mitmproxy](https://mitmproxy.org)). you need to disable ssl pinning to see requests. there will be no assistance provided by cobalt for this.
+
+example config (replace **ALL** values with ones you got from mitm):
+```
+'{
+    "iid": "<install_id here>",
+    "device_id": "<device_id here>",
+    "channel": "googleplay",
+    "app_name": "musical_ly",
+    "version_code": "310503",
+    "device_platform": "android",
+    "device_type": "Redmi+7",
+    "os_version": "13"
+}'
+```
+
+you can compress the json to save space. if you're using a `.env` file then the line would would look like this (***note the quotes***):
+```
+TIKTOK_DEVICE_INFO='{"iid":"<install_id here>","device_id":"<device_id here>","channel":"googleplay","app_name":"musical_ly","version_code":"310503","device_platform":"android","device_type":"Redmi+7","os_version":"13"}'
+```
+
 ### variables for web
 | variable name        | default              | example                 | description                                                                           |
 |:---------------------|:---------------------|:------------------------|:--------------------------------------------------------------------------------------|
@@ -72,4 +95,4 @@ sudo service nscd start
 | `IS_BETA`            | `0`                  | `1`                     | toggles beta tag next to cobalt logo. <br> `0`: disabled. `1`: enabled.               |
 | `PLAUSIBLE_HOSTNAME` | ➖                   | `plausible.io`*         | enables plausible analytics with provided hostname as receiver backend.               |
 
-\* don't use plausible.io as receiver backend unless you paid for their cloud service. use your own domain when hosting community edition of plausible. refer to their [docs](https://plausible.io/docs) when needed.
+\* don't use plausible.io as receiver backend unless you paid for their cloud service. use your own domain when hosting community edition of plausible. refer to their [docs](https://plausible.io/docs) when needed.

package.json

@@ -1,7 +1,7 @@
 {
     "name": "cobalt",
     "description": "save what you love",
-    "version": "7.12.1",
+    "version": "7.12.4",
     "author": "wukko",
     "exports": "./src/cobalt.js",
     "type": "module",
@@ -40,6 +40,6 @@
         "set-cookie-parser": "2.6.0",
         "undici": "^6.7.0",
         "url-pattern": "1.0.3",
-        "youtubei.js": "^9.1.0"
+        "youtubei.js": "^9.2.0"
     }
 }

src/front/cobalt.css

@@ -262,9 +262,8 @@ button:active,
     align-items: center;
 }
 #logo {
-    text-align: left;
+    text-align: center;
     font-size: 1rem;
-    white-space: nowrap;
     height: 2.5rem;
     align-items: center;
     display: flex;
@@ -452,6 +451,7 @@ button:active,
     display: flex;
     flex-direction: column;
     gap: 18px;
+    width: 100%;
 }
 .popup.small.visible {
     transform: translate(-50%, -50%);
@@ -799,7 +799,7 @@ button:active,
 }
 #cobalt-main-box #bottom button {
     width: auto;
-    padding: var(--gap) 1rem;
+    padding: var(--gap) 0.9rem;
 }
 .collapse-list {
     background: var(--subbackground);
@@ -990,47 +990,47 @@ button:active,
 .text-to-copy,
 .text-to-copy.text-backdrop,
 #filename-preview {
-    border-radius: 8px / 9px;
+    border-radius: 9px;
 }
 [type=checkbox] {
-    border-radius: 3px / 4px;
+    border-radius: 4px;
 }
 .popup,
 .scrollable .popup-content {
     border-radius: 12px;
 }
 .popup-header .glass-bkg {
-    border-top-left-radius: 11px 12px;
+    border-top-left-radius: 12px;
-    border-top-right-radius: 11px 12px;
+    border-top-right-radius: 12px;
     border-bottom: var(--accent-highlight) solid 0.1rem;
     top: -1px;
 }
 .popup-tabs .glass-bkg {
-    border-bottom-left-radius: 11px 12px;
+    border-bottom-left-radius: 12px;
-    border-bottom-right-radius: 11px 12px;
+    border-bottom-right-radius: 12px;
     border-top: var(--accent-highlight) solid 0.1rem;
     bottom: -1px;
 }
 .switches .switch:first-child {
-    border-top-left-radius: 8px 9px;
+    border-top-left-radius: 9px;
-    border-bottom-left-radius: 8px 9px;
+    border-bottom-left-radius: 9px;
 }
 .switches .switch:last-child {
-    border-top-right-radius: 8px 9px;
+    border-top-right-radius: 9px;
-    border-bottom-right-radius: 8px 9px;
+    border-bottom-right-radius: 9px;
 }
 .text-backdrop {
-    border-radius: 4px / 5px;
+    border-radius: 4px;
 }
 .collapse-list:first-child,
 .collapse-list:first-child .collapse-header {
-    border-top-left-radius: 7px 8px;
+    border-top-left-radius: 8px;
-    border-top-right-radius: 7px 8px;
+    border-top-right-radius: 8px;
 }
 .collapse-list:last-child,
 .collapse-list:last-child .collapse-header {
-    border-bottom-left-radius: 7px 8px;
+    border-bottom-left-radius: 8px;
-    border-bottom-right-radius: 7px 8px;
+    border-bottom-right-radius: 8px;
 }
 .collapse-list:last-child.expanded .collapse-header {
     border-radius: 0;

src/front/cobalt.js

@@ -22,7 +22,6 @@ const switchers = {
 };
 const checkboxes = [
     "alwaysVisibleButton",
-    "disableChangelog",
     "downloadPopup",
     "fullTikTokAudio",
     "muteAudio",
@@ -171,17 +170,21 @@ function notificationCheck(type) {
         default:
             changed = false;
     }
-    if (changed && sGet("changelogStatus") === `${version}` || type === "disable") {
+    if (changed && sGet("changelogStatus") === `${version}`) {
         setTimeout(() => {
             eid("about-footer").innerHTML = eid("about-footer").innerHTML.replace(notification, '');
             eid("tab-button-about-changelog").innerHTML = eid("tab-button-about-changelog").innerHTML.replace(notification, '')
         }, 900)
     }
-    if (sGet("disableChangelog") !== "true") {
+    if (!sGet("seenAbout") && !eid("about-footer").innerHTML.includes(notification)) {
-        if (!sGet("seenAbout") && !eid("about-footer").innerHTML.includes(notification)) eid("about-footer").innerHTML = `${notification}${eid("about-footer").innerHTML}`;
+        eid("about-footer").innerHTML = `${notification}${eid("about-footer").innerHTML}`;
-        if (sGet("changelogStatus") !== `${version}`) {
+    }
-            if (!eid("about-footer").innerHTML.includes(notification)) eid("about-footer").innerHTML = `${notification}${eid("about-footer").innerHTML}`;
+    if (sGet("changelogStatus") !== `${version}`) {
-            if (!eid("tab-button-about-changelog").innerHTML.includes(notification)) eid("tab-button-about-changelog").innerHTML = `${notification}${eid("tab-button-about-changelog").innerHTML}`;
+        if (!eid("about-footer").innerHTML.includes(notification)) {
+            eid("about-footer").innerHTML = `${notification}${eid("about-footer").innerHTML}`;
+        }
+        if (!eid("tab-button-about-changelog").innerHTML.includes(notification)) {
+            eid("tab-button-about-changelog").innerHTML = `${notification}${eid("tab-button-about-changelog").innerHTML}`;
         }
     }
 }
@@ -298,7 +301,6 @@ function checkbox(action) {
         case "reduceTransparency": eid("cobalt-body").classList.toggle('no-transparency'); break;
         case "disableAnimations": eid("cobalt-body").classList.toggle('no-animation'); break;
     }
-    action === "disableChangelog" && sGet(action) === "true" ? notificationCheck("disable") : notificationCheck();
 }
 function changeButton(type, text) {
     switch (type) {

src/localization/languages/en.json

@@ -1,7 +1,7 @@
 {
     "name": "english",
     "substrings": {
-        "ContactLink": "check the <a class=\"text-backdrop link\" href=\"{statusPage}\" target=\"_blank\">status page</a> or <a class=\"text-backdrop link\" href=\"{repo}\" target=\"_blank\">create an issue on github</a>."
+        "ContactLink": "check the <a class=\"text-backdrop link\" href=\"{statusPage}\" target=\"_blank\">status page</a> or <a class=\"text-backdrop link\" href=\"{repo}\" target=\"_blank\">create an issue on github</a>"
     },
     "strings": {
         "AppTitleCobalt": "cobalt",
@@ -80,7 +80,6 @@
         "Miscellaneous": "miscellaneous",
         "ModeToggleAuto": "auto",
         "ModeToggleAudio": "audio",
-        "SettingsDisableNotifications": "hide notifications",
         "MediaPickerTitle": "pick what to save",
         "MediaPickerExplanationPC": "click or right click to download what you want.",
         "MediaPickerExplanationPhone": "press or press and hold to download what you want.",

src/localization/languages/ru.json

@@ -80,7 +80,6 @@
         "Miscellaneous": "разное",
         "ModeToggleAuto": "авто",
         "ModeToggleAudio": "аудио",
-        "SettingsDisableNotifications": "cкрыть уведомления",
         "MediaPickerTitle": "выбери, что сохранить",
         "MediaPickerExplanationPC": "кликни то, что хочешь скачать. также можно скачать правой кнопки мыши.",
         "MediaPickerExplanationPhone": "нажми, или нажми и удерживай, чтобы скачать.",

src/modules/pageRender/elements.js

@@ -217,7 +217,7 @@ export function celebrationsEmoji() {
 export function urgentNotice(obj) {
     if (obj.visible) {
         return `<div id="urgent-notice" class="urgent-notice explanation">` +
-            `<span class="urgent-text" onclick="${obj.action}">${emoji(obj.emoji, 18)} ${obj.text}</span>` +
+            `<span id="urgent-notice-child" class="urgent-text" onclick="${obj.action}">${emoji(obj.emoji, 18)} ${obj.text}</span>` +
         `</div>`
     }
     return ``

src/modules/pageRender/page.js

@@ -523,10 +523,6 @@ export default function(obj) {
                     }, {
                         action: "disableMetadata",
                         name: t("SettingsDisableMetadata")
-                    }, {
-                        action: "disableChangelog",
-                        name: t("SettingsDisableNotifications"),
-                        padding: "no-margin"
                     }])
                 })
             }]

src/modules/processing/match.js

@@ -83,10 +83,8 @@ export default async function(host, patternMatch, url, lang, obj) {
                     id: patternMatch.id
                 });
                 break;
-            case "douyin":
             case "tiktok":
                 r = await tiktok({
-                    host: host,
                     postId: patternMatch.postId,
                     id: patternMatch.id,
                     fullAudio: obj.isTTFullAudio,

src/modules/processing/services/tiktok.js

@@ -1,42 +1,21 @@
 import { genericUserAgent } from "../../config.js";
 
-const userAgent = genericUserAgent.split(' Chrome/1')[0],
+const shortDomain = "https://vt.tiktok.com/";
-    config = {
+const apiPath = "https://api22-normal-c-alisg.tiktokv.com/aweme/v1/feed/?region=US&carrier_region=US";
-    tiktok: {
+const apiUserAgent = "TikTok/338014 CFNetwork/1410.1 Darwin/22.6.0";
-        short: "https://vt.tiktok.com/",
-        api: "https://api22-normal-c-useast2a.tiktokv.com/aweme/v1/feed/?aweme_id={postId}&version_code=262&app_name=musical_ly&channel=App&device_id=null&os_version=14.4.2&device_platform=iphone&device_type=iPhone9&region=US&carrier_region=US",
-        userAgent: "TikTok 26.2.0 rv:262018 (iPhone; iOS 14.4.2; en_US) Cronet"
-    },
-    douyin: {
-        short: "https://v.douyin.com/",
-        api: "https://www.iesdouyin.com/aweme/v1/web/aweme/detail/?aweme_id={postId}",
-        userAgent: "TikTok 26.2.0 rv:262018 (iPhone; iOS 14.4.2; en_US) Cronet"
-    }
-}
-
-function selector(j, h, id) {
-    if (!j) return false;
-    let t;
-    switch (h) {
-        case "tiktok":
-            t = j.aweme_list.filter(v => v.aweme_id === id)[0];
-            break;
-        case "douyin":
-            t = j.aweme_detail;
-            break;
-    }
-    if (t?.length < 3) return false;
-    return t;
-}
 
 export default async function(obj) {
     let postId = obj.postId ? obj.postId : false;
 
+    if (!process.env.TIKTOK_DEVICE_INFO) return { error: 'ErrorCouldntFetch' };
+
     if (!postId) {
-        let html = await fetch(`${config[obj.host].short}${obj.id}`, {
+        let html = await fetch(`${shortDomain}${obj.id}`, {
             redirect: "manual",
-            headers: { "user-agent": userAgent }
+            headers: {
-        }).then((r) => { return r.text() }).catch(() => { return false });
+                "user-agent": genericUserAgent.split(' Chrome/1')[0]
+            }
+        }).then(r => r.text()).catch(() => {});
 
         if (!html) return { error: 'ErrorCouldntFetch' };
 
@@ -48,30 +27,35 @@ export default async function(obj) {
     }
     if (!postId) return { error: 'ErrorCantGetID' };
 
-    let detail;
+    let deviceInfo = JSON.parse(process.env.TIKTOK_DEVICE_INFO);
-    detail = await fetch(config[obj.host].api.replace("{postId}", postId), {
+        deviceInfo = new URLSearchParams(deviceInfo).toString();
-        headers: {
-            "user-agent": config[obj.host].userAgent
-        }
-    }).then((r) => { return r.json() }).catch(() => { return false });
 
-    detail = selector(detail, obj.host, postId);
+    let apiURL = new URL(apiPath);
+        apiURL.searchParams.append("aweme_id", postId);
+
+    let detail = await fetch(`${apiURL.href}&${deviceInfo}`, {
+        headers: {
+            "user-agent": apiUserAgent
+        }
+    }).then(r => r.json()).catch(() => {});
+
+    detail = detail?.aweme_list?.find(v => v.aweme_id === postId);
     if (!detail) return { error: 'ErrorCouldntFetch' };
 
     let video, videoFilename, audioFilename, isMp3, audio, images,
-        filenameBase = `${obj.host}_${detail.author.unique_id}_${postId}`;
+        filenameBase = `tiktok_${detail.author.unique_id}_${postId}`;
 
-    if (obj.host === "tiktok") {
+    images = detail.image_post_info?.images;
-        images = detail.image_post_info ? detail.image_post_info.images : false
+
-    } else {
+    let playAddr = detail.video.play_addr_h264;
-        images = detail.images ? detail.images : false
+
-    }
+    if (!playAddr) playAddr = detail.video.play_addr;
 
     if (!obj.isAudioOnly && !images) {
-        video = detail.video.play_addr.url_list[0];
+        video = playAddr.url_list[0];
         videoFilename = `${filenameBase}.mp4`;
     } else {
-        let fallback = detail.video.play_addr.url_list[0];
+        let fallback = playAddr.url_list[0];
         audio = fallback;
         audioFilename = `${filenameBase}_audio_fv`;  // fv - from video
         if (obj.fullAudio || fallback.includes("music")) {
@@ -94,7 +78,7 @@ export default async function(obj) {
     if (images) {
         let imageLinks = [];
         for (let i in images) {
-            let sel = obj.host === "tiktok" ? images[i].display_image.url_list : images[i].url_list;
+            let sel = images[i].display_image.url_list;
             sel = sel.filter(p => p.includes(".jpeg?"))
             imageLinks.push({url: sel[0]})
         }

src/modules/processing/servicesConfig.json

@@ -22,6 +22,7 @@
             "patterns": [
                 ":user/status/:id",
                 ":user/status/:id/video/:index",
+                ":user/status/:id/photo/:index",
                 ":user/status/:id/mediaviewer",
                 ":user/status/:id/mediaViewer"
             ],
@@ -79,7 +80,7 @@
             "alias": "instagram reels, posts & stories",
             "altDomains": ["ddinstagram.com", "d.ddinstagram.com", "g.ddinstagram.com"],
             "patterns": [
-                "reels/:postId", "reel/:postId", "p/:postId", ":username/p/:postId",
+                "reels/:postId", ":username/reel/:postId", "reel/:postId", "p/:postId", ":username/p/:postId",
                 "tv/:postId", "stories/:username/:storyId"
             ],
             "enabled": true