Commit graph

1595 commits

Author SHA1 Message Date
Tobi
3d314169b9
Merge pull request #943 from TeamNewPipe/fix/sc/comments
[SoundCloud] Fix getting more comments
2022-10-29 22:19:50 +02:00
AudricV
c230d84df1
Fix Checkstyle error in YoutubeCommentsInfoItemExtractor 2022-10-29 13:24:19 +02:00
xz-dev
0ffcb32d9c
[YouTube] Add comment reply count support (#936)
Add comment reply count support for YouTube and introduce `CommentsInfoItem.UNKNOWN_REPLY_COUNT` constant

Co-authored-by: AudricV <74829229+AudricV@users.noreply.github.com>
Co-authored-by: Tobi <TobiGr@users.noreply.github.com>
2022-10-15 12:40:06 +02:00
TobiGr
4d136599bd [SoundCloud] Fix getting more comments 2022-10-11 15:44:54 +02:00
Tobi
a822e91909
Merge pull request #939 from TurtleArmyMc/fix/SoundcloudPlaylistExtractor_track_order
Fix SoundcloudPlaylistExtractor: tracks are in correct order
2022-10-10 22:28:18 +02:00
TobiGr
02810a7db7 Add a comment 2022-10-10 22:22:12 +02:00
Tobi
54092fc3c7
Merge pull request #930 from Isira-Seneviratne/Avoid_Comparator_NPE
Avoid possible NullPointerException in MediaCCCRecentKiosk.
2022-10-10 11:43:34 +02:00
TurtleArmyMc
bf70d32eb4 Fix SoundcloudPlaylistExtractor: tracks are in correct order 2022-09-30 15:26:25 -04:00
AudricV
8067c43837
[YouTube] Don't use a specific letter for the decryption function name pattern
Use the same possible characters for variables everywhere, in order to avoid
potential future throttling parameter decryption function name parsing issues
related to the usage of other letter(s) than b.
2022-09-24 21:49:22 +02:00
AudricV
abcee87167
[YouTube] Fix throttling parameter decryption function regex
- Quote the function name, as it may contain special regex symbols, such as
  dollar;
- Support multiple lines;
- Use what looks like the end of the function for the end of the regex (this
  part is inspired from yt-dlp throttling parameter decryption regex);
- Move the throttling function body regex into a private and static constant.
2022-09-24 21:28:09 +02:00
ThetaDev
7244be7627
[YouTube] Add JavaScript lexer to parse completely throttling decryption function (#905) 2022-09-24 19:55:46 +02:00
Isira Seneviratne
0e31c86aee Avoid possible NullPointerException in MediaCCCRecentKiosk. 2022-09-21 05:40:07 +05:30
Stypox
a99af9bb6e
Merge pull request #927 from Stypox/fix-feed-thumbnail
[YouTube] Return mqdefault thumbnails in fast feed
2022-09-19 08:52:01 +02:00
Stypox
93ef73e2f2
[YouTube] Return mqdefault thumbnails in feed
The hqdefault thumbnails, which are used by default, have black bars at the top and at the bottom, and are thus not optimal.
2022-09-13 16:46:54 +02:00
AudricV
d8ddeb549c
[YouTube] Support new trending structure and filter recently trending section
This new structure allow us to filter easily Trending shorts and Recently
trending sections.

On the previous one, this Recently trending section is now filtered, by
checking whether sections have a title, which isn't the case for normal trends
contrary to the other ones.

This makes that the extractor returns now only the real 50 "Now" YouTube
trends.

Elements inside arrays are now extracted dynamically instead of only the ones
of the first index, using Java 8's Stream API.

The getInitialPage() method of YoutubeTrendingExtractor can now throw a
ParsingException if no selected tab (corresponding to the one of the trends
type extracted) has been found.

Finally, the licence header has been moved to the top of the file and updated.
2022-09-11 18:52:12 +02:00
AudricV
884da40f65
Merge pull request #926 from AudricV/fix-yt-like-count-extraction
[YouTube] Fix extraction of like count with the new data model
2022-09-11 12:21:38 +02:00
AudricV
119b9129e2
[YouTube] Fix extraction of like count with the new data model
In the new data model currently A/B tested or deployed, the like count button
data is the same, but its path has been changed.

The extraction of the like count has been also improved, by using the multiple
accessibility data available instead of the only one which was used before.
This accessibility data has been also deprioritized, because it is language
dependent when there is no view.

Like count is always known, even for age-restricted videos, so returning -1 if
the like count extraction fails on this type of videos is not needed and hides
an extraction error: that's the reason why this return has been removed and the
exception will always be thrown, even if a video is age-restricted.
2022-09-10 23:28:38 +02:00
ThetaDev
4905f74700
[YouTube] Fix throttling parameter decryption on Android
Escape the curly brace in the regular expression used to parse the throttling
parameter decryption function to allow its compatibility on Android.
2022-09-10 17:03:39 +02:00
Stypox
fa25a7280f
Downgrade Rhino since it uses a class not available on Android
Reverts #774, see TeamNewPipe/NewPipe#8876
2022-08-25 09:49:13 +02:00
Stypox
25f0ce7305
Merge pull request #911 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.15.3
Bump jsoup from 1.15.2 to 1.15.3
2022-08-24 14:23:44 +02:00
dependabot[bot]
1d7a3a90a4
Bump jsoup from 1.15.2 to 1.15.3
Bumps [jsoup](https://github.com/jhy/jsoup) from 1.15.2 to 1.15.3.
- [Release notes](https://github.com/jhy/jsoup/releases)
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES)
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.15.2...jsoup-1.15.3)

---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-08-24 09:14:26 +00:00
Isira Seneviratne
41254ae12a Remove annotation. 2022-08-24 06:59:17 +05:30
Isira Seneviratne
943b7c033b Remove EMPTY_STRING. 2022-08-24 06:59:17 +05:30
litetex
d6577e5e0b
Merge pull request #882 from litetex/fix-all-tests
Fix all tests
2022-08-23 16:19:27 +02:00
AudricV
03d9a4fe9d
Fix typos in YoutubeStreamExtractor.tryDecryptUrl 2022-08-21 21:17:03 +02:00
litetex
0beb55a232 Improve `YoutubeThrottlingDecrypterTest` 2022-08-21 20:17:05 +02:00
litetex
9e93d6b193 YoutubeThrottlingDecrypter: Check if returned string is a valid JavaScript function 2022-08-21 20:16:45 +02:00
litetex
ed0a07af4e YoutubeThrottlingDecrypter: Patch regex 2022-08-21 20:15:53 +02:00
litetex
641a447d9b Updated consent-related mock-data 2022-08-21 18:43:48 +02:00
litetex
8ff7a90f52 Improved consent cookie related constants and documentation 2022-08-21 18:41:40 +02:00
AudricV
cb64a480cd
[YouTube] Remove deprecated code in YoutubeThrottlingDecrypter
YoutubeThrottlingDecrypter is now a non-instantiable final class and non-static
attributes have been removed.

The static attributes of this class have been renamed, in order to respect the
naming specification used.
2022-08-16 15:01:05 +02:00
litetex
5bbea3a8f2 Fix `YoutubeMixPlaylistExtractorTest` again
Doesn't work for kurgesagt videos anymore, used music video (copyright free) instead
2022-08-14 15:12:27 +02:00
litetex
da06166065 Updated mock data of `YoutubeSearchExtractorTest$Suggestion` 2022-08-14 15:12:26 +02:00
litetex
2e36ab1578 Fixed `YoutubeSearchExtractorTest$Suggestion` 2022-08-14 15:12:26 +02:00
litetex
2a8a623643 `YoutubePlaylistLinkHandlerFactoryTest`: Use parameterized tests 2022-08-14 14:48:31 +02:00
litetex
938e69a16a Fixed `YoutubePlaylistLinkHandlerFactoryTest` 2022-08-14 14:48:30 +02:00
litetex
844de3e378 Fix `YoutubeStreamLinkHandlerFactoryTest ` and parameterized the tests
used code from #833
2022-08-14 14:48:30 +02:00
litetex
c1a72b8240 Use Android Studios code style 2022-08-14 14:48:29 +02:00
litetex
fc27b8a5b8 Use preexisting `ContentNotAvailableException` 2022-08-14 14:48:29 +02:00
Stypox
17bad6cedf Fix test about exception type: NPE, not IllegalArgment
The relevant change was made in #877
2022-08-14 14:48:28 +02:00
litetex
ecfc370685 Fixed all YTMixPlaylists
Added option to choose if you want to consent or not - currently this is done by a static variable in ``YoutubeParsingHelper`` - may not be the best long-term solution but for now the tests work again (in EU countries) 🥳
2022-08-14 14:48:27 +02:00
litetex
2b6fe294b2 Fixed `YoutubeMusicSearchExtractorTest`
The renderers and queries got changed.
2022-08-14 14:48:27 +02:00
litetex
d6586da614 Failing test works locally without any problems. Reformatted it a bit. 2022-08-14 14:48:27 +02:00
litetex
bf3ae5e679 Fixed `SoundcloudStreamLinkHandlerFactoryTest`
* Removed the unnecessary ``public``
* Use parameterized tests
* One song got removed (which caused the test failure), replaced it with another song
2022-08-14 14:48:27 +02:00
litetex
504f81036e Fixed `PeertubePlaylistExtractorTest`
Also removed the unnecessary ``public`` and reformatted the code a bit
2022-08-14 14:48:26 +02:00
litetex
e7c12258f4 Fixed `BandcampSearchExtractorTest`
Also removed the unnecessary ``public``
2022-08-14 14:48:26 +02:00
litetex
9e8724df4d Fixed `YoutubeStreamExtractorLivestreamTest`
The "Lofi Girl"-stream got interrupted by a copyright strike and had to be restarted. Because of this a new id is now used.
2022-08-14 14:48:26 +02:00
AudricV
472f5d9e9c
[YouTube] Update mocks 2022-08-12 19:20:32 +02:00
AudricV
7bdca33a87
[YouTube] Ensure that an additional player response is the correct one
If YouTube detect that requests come from a third party client, they may
replace the real player response by another one of a video saying that this
content is not available on this app and to watch it on the latest version of
YouTube. We can detect this by checking whether the video ID of the player
response returned is the same as the one requested by the extractor.
2022-08-12 19:20:31 +02:00
AudricV
c82317e318
[YouTube] Spoof more mobile clients
Additional parameters have been added to the player requests of ANDROID and IOS
clients:

- for both clients: osName and osVersion: their respective values are:
  - for the ANDROID one: Android and 12;
  - for the IOS one: iOS and 15.6.0.19G71.
- for the ANDROID client: androidTargetSdkVersion, with the Android SDK version
  corresponding to the Android version used in the player requests of this
  client. This parameter is now required with this client to be sure to get a
  correct player response, otherwise, the one of a video saying that this
  content is not available in this app and to watch it with the latest version
  of YouTube can be returned instead;
- for the IOS client: deviceMake, with Apple as its value.

The iOS version sent in the IOS client player requests has been also updated to
the version 15.6 of the OS.

Finally, a comment about the requirement to use the signature timestamp from
the player JavaScript base file for HTML5 player requests on videos with
obfuscated URLs has been added and replaces a previous one which may be not
true.
2022-08-12 19:20:31 +02:00