CypherpunkSamurai/NewPipeExtractor
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity
2299 commits 2 branches 0 tags 95 MiB
Commit graph

2299 commits

This branch
This branch
All branches
Author SHA1 Message Date
litetex
c1a72b8240 Use Android Studios code style 2022-08-14 14:48:29 +02:00
litetex
fc27b8a5b8 Use preexisting `ContentNotAvailableException` 2022-08-14 14:48:29 +02:00
Stypox
17bad6cedf Fix test about exception type: NPE, not IllegalArgment 
The relevant change was made in #877
 2022-08-14 14:48:28 +02:00
litetex
ecfc370685 Fixed all YTMixPlaylists 
Added option to choose if you want to consent or not - currently this is done by a static variable in ``YoutubeParsingHelper`` - may not be the best long-term solution but for now the tests work again (in EU countries) 🥳
 2022-08-14 14:48:27 +02:00
litetex
2b6fe294b2 Fixed `YoutubeMusicSearchExtractorTest` 
The renderers and queries got changed.
 2022-08-14 14:48:27 +02:00
litetex
d6586da614 Failing test works locally without any problems. Reformatted it a bit. 2022-08-14 14:48:27 +02:00
litetex
bf3ae5e679 Fixed `SoundcloudStreamLinkHandlerFactoryTest` 
* Removed the unnecessary ``public``
* Use parameterized tests
* One song got removed (which caused the test failure), replaced it with another song
 2022-08-14 14:48:27 +02:00
litetex
504f81036e Fixed `PeertubePlaylistExtractorTest` 
Also removed the unnecessary ``public`` and reformatted the code a bit
 2022-08-14 14:48:26 +02:00
litetex
e7c12258f4 Fixed `BandcampSearchExtractorTest` 
Also removed the unnecessary ``public``
 2022-08-14 14:48:26 +02:00
litetex
9e8724df4d Fixed `YoutubeStreamExtractorLivestreamTest` 
The "Lofi Girl"-stream got interrupted by a copyright strike and had to be restarted. Because of this a new id is now used.
 2022-08-14 14:48:26 +02:00
Tobi
76aad92fa5
Merge pull request #890 from AudricV/yt-clients-update-improvements-and-extraction-fixes 
[YouTube] Update clients version and improve extraction of API key and client version of WEB client
 2022-08-12 23:32:41 +02:00
AudricV
472f5d9e9c
[YouTube] Update mocks 2022-08-12 19:20:32 +02:00
AudricV
7bdca33a87
[YouTube] Ensure that an additional player response is the correct one 
If YouTube detect that requests come from a third party client, they may
replace the real player response by another one of a video saying that this
content is not available on this app and to watch it on the latest version of
YouTube. We can detect this by checking whether the video ID of the player
response returned is the same as the one requested by the extractor.
 2022-08-12 19:20:31 +02:00
AudricV
c82317e318
[YouTube] Spoof more mobile clients 
Additional parameters have been added to the player requests of ANDROID and IOS
clients:

- for both clients: osName and osVersion: their respective values are:
  - for the ANDROID one: Android and 12;
  - for the IOS one: iOS and 15.6.0.19G71.
- for the ANDROID client: androidTargetSdkVersion, with the Android SDK version
  corresponding to the Android version used in the player requests of this
  client. This parameter is now required with this client to be sure to get a
  correct player response, otherwise, the one of a video saying that this
  content is not available in this app and to watch it with the latest version
  of YouTube can be returned instead;
- for the IOS client: deviceMake, with Apple as its value.

The iOS version sent in the IOS client player requests has been also updated to
the version 15.6 of the OS.

Finally, a comment about the requirement to use the signature timestamp from
the player JavaScript base file for HTML5 player requests on videos with
obfuscated URLs has been added and replaces a previous one which may be not
true.
 2022-08-12 19:20:31 +02:00
AudricV
d0549a5a52
[YouTube] Update client versions and use a real version for the iOS client 
The iOS version can be got easily in fact, by looking at the What's New section of the App Store' app page.
 2022-08-12 19:20:31 +02:00
AudricV
d7e678aca2
[YouTube] Improve WEB client version and API key HTML extraction 
Common code in WEB client version HTML extraction has been deduplicated, usage of the Java 8 Stream API has been made and initial data fallback has been used as a last resort.
This means that the client version extraction from regexes will be used before this fallback, as it doesn't contain the full client version.
This can be used as a way to fingerprint the extractor, even if it seems to be not the case.
 2022-08-12 19:20:30 +02:00
AudricV
6a885ef5ab
Merge pull request #891 from Theta-Dev/fix-throttling-decrypter 
[YouTube] Fix extraction of more complex nsig functions
 2022-08-12 18:24:21 +02:00
AudricV
5b548340e8
[YouTube] Catch any exception in YoutubeThrottlingDecrypter.apply and improve docs 
This will prevent any future extractor break due to decryption failure, like it was excepted to be the case before.

Some documentation about the throttling decryption has been also improved.
 2022-08-12 17:49:36 +02:00
ThetaDev
52ded6e3d7
Handle curly braces inside strings in StringUtils.matchToClosingParenthesis 
This is required to extract fully more complex YouTube nsig functions.
 2022-08-12 16:32:00 +02:00
Stypox
d12003651b
Merge pull request #886 from Isira-Seneviratne/toArray_improvements 
Make improvements to methods using toArray().
 2022-08-06 22:34:23 +02:00
Isira Seneviratne
7daca10a06 Make improvements to methods using toArray(). 2022-08-06 05:21:12 +05:30
Stypox
2906be22af
Merge pull request #881 from Isira-Seneviratne/String_join 
Use String.join() and Collectors.joining().
 2022-08-04 12:09:33 +02:00
Stypox
4ddb96a86f
Merge pull request #883 from TeamNewPipe/dependabot/gradle/com.google.code.gson-gson-2.9.1 
Bump gson from 2.9.0 to 2.9.1
 2022-08-04 11:20:41 +02:00
Isira Seneviratne
64771c5712 Use String.join() and Collectors.joining(). 2022-08-04 05:18:13 +05:30
Stypox
fc8b5ebbc6
Merge pull request #878 from Isira-Seneviratne/Use_Collections 
Use Collections methods.
 2022-08-03 22:50:41 +02:00
Stypox
4a4939d89c
Merge pull request #877 from Isira-Seneviratne/Use_Objects_requireNonNull 
Use Objects.requireNonNull().
 2022-08-03 22:45:35 +02:00
Stypox
c336bd58a5
Merge pull request #879 from TeamNewPipe/dependabot/gradle/org.junit-junit-bom-5.9.0 
Bump junit-bom from 5.8.2 to 5.9.0
 2022-08-03 21:06:20 +02:00
dependabot[bot]
325af31e5f
Bump gson from 2.9.0 to 2.9.1 
Bumps [gson](https://github.com/google/gson) from 2.9.0 to 2.9.1.
- [Release notes](https://github.com/google/gson/releases)
- [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md)
- [Commits](https://github.com/google/gson/compare/gson-parent-2.9.0...gson-parent-2.9.1)

---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-08-01 09:26:14 +00:00
dependabot[bot]
d905636021
Bump junit-bom from 5.8.2 to 5.9.0 
Bumps [junit-bom](https://github.com/junit-team/junit5) from 5.8.2 to 5.9.0.
- [Release notes](https://github.com/junit-team/junit5/releases)
- [Commits](https://github.com/junit-team/junit5/compare/r5.8.2...r5.9.0)

---
updated-dependencies:
- dependency-name: org.junit:junit-bom
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-07-27 09:20:12 +00:00
Isira Seneviratne
1af6b8eedb Use Collections.singletonList(). 2022-07-27 07:35:57 +05:30
Isira Seneviratne
ff60e05c76 Use Collections.singletonMap(). 2022-07-27 07:35:57 +05:30
Isira Seneviratne
682a4263e5 Use Objects.requireNonNull(). 2022-07-27 06:55:26 +05:30
Stypox
8c5f014a6f
Merge pull request #869 from mhmdanas/add-workflow-permissions 
Use minimum required permissions for GitHub workflows
 2022-07-13 18:58:32 +02:00
Stypox
954a294e27
Merge pull request #870 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.15.2 
Bump jsoup from 1.15.1 to 1.15.2
 2022-07-13 17:48:06 +02:00
Mohammed Anas
f57049d2c0
Use temurin instead of adopt (#868) 
See
https://blog.adoptopenjdk.net/2021/08/goodbye-adoptopenjdk-hello-adoptium/.
 2022-07-10 18:38:08 +03:00
Stypox
5bd7bf20cc
Merge pull request #867 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.7.1 
Bump spotbugs-annotations from 4.7.0 to 4.7.1
 2022-07-06 14:23:44 +02:00
Stypox
5ab74b3631
Merge pull request #857 from FireMasterK/video-title 
Get original untranslated title for YouTube
 2022-07-06 10:26:45 +02:00
dependabot[bot]
122365005a
Bump jsoup from 1.15.1 to 1.15.2 
Bumps [jsoup](https://github.com/jhy/jsoup) from 1.15.1 to 1.15.2.
- [Release notes](https://github.com/jhy/jsoup/releases)
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES)
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.15.1...jsoup-1.15.2)

---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-07-04 09:38:16 +00:00
mhmdanas
414186cff5 Use minimum required permissions for GitHub workflows 
This reduces the attack surface if the workflows are ever compromised.
 2022-07-03 23:55:28 +03:00
dependabot[bot]
e9b4be3e3c
Bump spotbugs-annotations from 4.7.0 to 4.7.1 
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.7.0 to 4.7.1.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.7.0...4.7.1)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-06-28 09:15:07 +00:00
Stypox
5219a705ba
Merge pull request #864 from AudricV/fetch-android-client-for-ended/post-livestreams 
[YouTube] Fetch the ANDROID client for ended/post livestreams
 2022-06-24 14:55:20 +02:00
AudricV
090debd83b
[YouTube] Fetch the ANDROID client for ended/post livestreams 
The ANDROID client was only fetched for video contents, where it can be useful on ended/post livestreams, if the n parameter of the WEB client cannot be decrypted, to avoid throttling issues (because the WEB client was only used before for ended/post livestreams).

It also provides an exclusive 48kbps M4A audio format in the adaptiveFormats array of the JSON player response, like other mobile clients (which can be also extracted from the response of the DASH manifest URL returned into the WEB client player's response, but the DASH manifest is not used by the extractor).

A note about non-fatality of fetching or parsing issues of the ANDROID and IOS clients has been added.
 2022-06-21 18:53:49 +02:00
litetex
a26bcc55c4
Merge pull request #845 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.7.0 
Bump spotbugs-annotations from 4.6.0 to 4.7.0
 2022-06-19 15:36:55 +02:00
dependabot[bot]
424eb1c559
Bump spotbugs-annotations from 4.6.0 to 4.7.0 
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.6.0 to 4.7.0.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.6.0...4.7.0)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-06-19 13:29:18 +00:00
litetex
2712c3d788
Merge pull request #847 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.15.1 
Bump jsoup from 1.14.3 to 1.15.1
 2022-06-19 15:28:43 +02:00
dependabot[bot]
281d2b9f81
Bump jsoup from 1.14.3 to 1.15.1 
Bumps [jsoup](https://github.com/jhy/jsoup) from 1.14.3 to 1.15.1.
- [Release notes](https://github.com/jhy/jsoup/releases)
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES)
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.14.3...jsoup-1.15.1)

---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-06-19 13:15:40 +00:00
litetex
f775155d25
Merge pull request #846 from litetex/remove-unused-methods 
Remove unused methods
 2022-06-19 15:12:15 +02:00
litetex
9d625dd75f
Merge pull request #860 from gliptak/patch-1 
Correct PeertubePlaylistExtractorTest unit tests
 2022-06-19 15:09:27 +02:00
Gábor Lipták
28c9340d69
Correct unit tests 2022-06-18 11:18:38 -04:00
Stypox
1b51eab664
Merge pull request #859 from AudricV/delivery-methods-fixes-and-improvements 
Fix extraction of some properties in ItagItems for YouTube livestreams and post-live streams and remove completely SoundCloud HLS workaround
 2022-06-17 15:46:13 +02:00