1362 commits

Author	SHA1	Message	Date
dependabot[bot]	ad403ce73e	Bump spotbugs-annotations from 4.0.2 to 4.2.3 ... Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.0.2 to 4.2.3. - [Release notes](https://github.com/spotbugs/spotbugs/releases) - [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md) - [Commits](https://github.com/spotbugs/spotbugs/compare/4.0.2...4.2.3) --- updated-dependencies: - dependency-name: com.github.spotbugs:spotbugs-annotations dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>	2021-06-23 09:57:42 +00:00
bopol	65fd8740aa	Merge pull request #658 from TeamNewPipe/revert-651-fix/streamType ... [YouTube] Fix getting stream type of live streams	2021-06-23 11:33:24 +02:00
bopol	361f142621	Use badge style instead of label	2021-06-23 11:25:59 +02:00
bopol	58f109ddec	adress stypox review	2021-06-23 11:15:40 +02:00
bopol	fe29c78fe1	Merge pull request #656 from TeamNewPipe/dependabot/gradle/org.mozilla-rhino-1.7.13 ... Bump rhino from 1.7.12 to 1.7.13	2021-06-20 00:46:42 +02:00
bopol	e9412fdadc	Merge pull request #654 from TeamNewPipe/dependabot/gradle/com.google.code.gson-gson-2.8.7 ... Bump gson from 2.8.6 to 2.8.7	2021-06-20 00:35:24 +02:00
TobiGr	1f9a99c931	[SoundCloud] Update HARDCODED_CLIENT_ID	2021-06-18 15:30:54 +02:00
Tobi	c998012c28	Revert "[YouTube] Fix getting stream type of live streams"	2021-06-18 09:20:32 +02:00
dependabot[bot]	38d0339753	Bump rhino from 1.7.12 to 1.7.13 ... Bumps [rhino](https://github.com/mozilla/rhino) from 1.7.12 to 1.7.13. - [Release notes](https://github.com/mozilla/rhino/releases) - [Changelog](https://github.com/mozilla/rhino/blob/master/RELEASE-NOTES.md) - [Commits](https://github.com/mozilla/rhino/commits) --- updated-dependencies: - dependency-name: org.mozilla:rhino dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>	2021-06-17 21:49:07 +00:00
dependabot[bot]	5a46878b09	Bump gson from 2.8.6 to 2.8.7 ... Bumps [gson](https://github.com/google/gson) from 2.8.6 to 2.8.7. - [Release notes](https://github.com/google/gson/releases) - [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md) - [Commits](https://github.com/google/gson/compare/gson-parent-2.8.6...gson-parent-2.8.7) --- updated-dependencies: - dependency-name: com.google.code.gson:gson dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>	2021-06-17 21:49:01 +00:00
TobiGr	6762fe3357	[YouTube] Fix getting stream type of live streams	2021-06-10 16:51:00 +02:00
TobiGr	f7f727d19c	Add constant NO_LIKE_COUNT to CommentsInfoItem	2021-06-08 10:28:44 +02:00
TobiGr	b70c0f93c7	Add streamPosition for comments ... SoundCloud is the only service which supports adding comments at a specific timestamp in the stream.	2021-06-08 10:27:52 +02:00
Tobi	d4186d100b	Merge pull request #580 from TeamNewPipe/accountTerminated ... Add AccountTerminatedException for terminated channels	2021-06-08 09:55:58 +02:00
TobiGr	89e0514d8b	Fix Javadoc generation fix	2021-06-08 09:30:05 +02:00
TobiGr	80d3052033	Fix Javadoc generation	2021-06-08 09:26:01 +02:00
TobiGr	fa444c8298	Remove assertations of exception messages	2021-06-06 11:38:43 +02:00
Stypox	2158ca4060	Remove two unneded warning suppressions	2021-06-05 14:15:48 +02:00
Stypox	13f192704d	Add utility method getService() to Info object	2021-06-05 14:14:19 +02:00
XiangRongLin	0ad51e76fb	Merge pull request #638 from TeamNewPipe/spelling ... Improve exception message thrown when not accepting an URL in LinkHandlerFactory#fromUrl(String url, String baseUrl)	2021-06-04 18:23:31 +02:00
XiangRongLin	faa503c7dd	Fix PeertubeAccountExtractorTest	2021-06-03 19:54:42 +02:00
TiA4f8R	ac31f3a883	Specify UTF-8 file encoding in RecordingDownloader and MockDownloader ... On Windows, mocks are recorded and read with the Cp1252 encoding so it breaks the mocks on non ASCII characters for Linux OS (and so the CI). The project is in Java 8, so we can't use FileReader(File, Charset) and FileReader(File, Charset) because these methods require Java 11. Instead of changing the Java version of the extractor, use OutputStreamWriter and FileOutputStream instead of FileWriter and InputStreamReader and FileInputStream instead of FileReader.	2021-06-03 18:38:25 +02:00
bopol	ff11c2df2a	Merge pull request #628 from litetex/fix-broken-yt-liked-comments ... Fix broken yt likes in comments	2021-05-29 11:04:11 +02:00
TobiGr	1ff56a85df	Update junit from 4.13.1 to 4.13.2 ... https://github.com/junit-team/junit4/blob/main/doc/ReleaseNotes4.13.2.md	2021-05-28 15:09:26 +02:00
TobiGr	858e007804	Improve exception message thrown when not accepting an URL in LinkHandlerFactory#fromUrl(String url, String baseUrl)	2021-05-28 14:59:44 +02:00
Tobi	519bba70b3	Merge pull request #630 from XiangRongLin/fix_tests ... Fix MetaInfoTest.expectedMetaInfo	2021-05-28 14:16:53 +02:00
litetex	289b84e68b	Removed unused code	2021-05-27 21:27:44 +02:00
litetex	e81b0e2885	Tweaked code	2021-05-27 19:48:31 +02:00
litetex	8c96545e57	Merge branch 'TeamNewPipe:dev' into fix-broken-yt-liked-comments	2021-05-24 18:06:45 +02:00
litetex	12fb18c310	Updated mock data (only the affected)	2021-05-24 18:05:26 +02:00
litetex	8e08a2aac7	Improved docs	2021-05-24 18:05:26 +02:00
litetex	020acfed71	Fixed tests and added new ones	2021-05-24 18:05:26 +02:00
litetex	b310922fc0	Better exception message	2021-05-24 18:05:25 +02:00
litetex	86bb9efb5d	Added likeCount back to tests	2021-05-24 18:05:25 +02:00
litetex	46654f5fca	Reformatted code	2021-05-24 18:05:25 +02:00
litetex	2174685c5c	Reimplemented likeCount	2021-05-24 18:05:25 +02:00
litetex	01cfb55505	Update YoutubeCommentsInfoItemExtractor.java	2021-05-24 18:05:25 +02:00
litetex	bedcd87abb	Fixed occurrences where voteCount is not set (<=0 likes)	2021-05-24 18:05:25 +02:00
litetex	545c0a6f42	Fixed comment	2021-05-24 18:05:24 +02:00
TiA4f8R	a00fdcbd3d	Format the indentation of the definition of the apiUrl string in SoundcloudParsingHelper.resolveFor method ... Co-authored-by: Tobi <TobiGr@users.noreply.github.com>	2021-05-23 20:27:45 +02:00
TiA4f8R	6b607eb38d	Update the hardcoded client_id	2021-05-23 18:08:44 +02:00
TiA4f8R	4552ea9c9f	Use the SOUNDCLOUD_API_V2_URL constant in all the SoundCloud package	2021-05-23 18:08:44 +02:00
TiA4f8R	86308d0603	Use a lightweight request to check if the client_id is valid ... Request the api-v2 host with the client_id instead of checking if the streams of a SoundCloud track are not empty: if it is valid, the API returns 404, otherwise it should return 401.	2021-05-23 18:08:43 +02:00
TiA4f8R	c5c190500c	Format the SoundCloud package of the extractor and fix some warnings ... Use final where possible in the package and format code to be in the 100 caracters per line limit. Fix some warnings generated by Android Studio and do some code improvements	2021-05-23 18:08:42 +02:00
Tobi	636e27333b	Merge pull request #603 from TeamNewPipe/peertube_hls ... Add PeerTube HLS streams	2021-05-23 11:49:40 +02:00
XiangRongLin	172394cd52	Update mocks	2021-05-22 20:33:20 +02:00
XiangRongLin	18c8cac40c	Fix MetaInfoTest.expectedMetaInfo ... Expected meta changed	2021-05-21 21:00:08 +02:00
Hassan Igbaria	e8b5364745	Update YoutubeStreamExtractor.java	2021-05-21 16:22:24 +03:00
litetex	c3b837fe3b	Deduped code; Using default values...	2021-05-20 20:22:40 +02:00
litetex	b934c7ccbb	Removed unused import	2021-05-20 20:11:35 +02:00
litetex	10cf081145	Removed likeCount and added textualVoteCount	2021-05-20 20:10:14 +02:00
TobiGr	1acc53b8db	[PeerTube] Add support for HLS streams	2021-05-15 12:39:23 +02:00
Tobi	3a3ade20f4	Merge pull request #620 from TeamNewPipe/bandcamp-comments ... Add Bandcamp comments (first page)	2021-05-02 13:43:53 +02:00
Fynn Godau	85a4270bd9	[Bandcamp] Fix comments test	2021-05-02 13:07:24 +02:00
TobiGr	ec50570053	Add mocks	2021-05-01 17:50:19 +02:00
TobiGr	f4404b5cc8	Add tests for more reasons	2021-05-01 17:50:19 +02:00
TobiGr	fc998589dc	Detect channels which have been terminated due to copyright infringement	2021-05-01 17:50:19 +02:00
TobiGr	bb3861ddce	Add AccountTerminatedException for better error handling of terminated channels	2021-05-01 17:50:19 +02:00
Tobi	284362fc15	Merge pull request #608 from TeamNewPipe/youtube-fix-segment-test ... [YouTube] fix tests	2021-05-01 17:24:41 +02:00
TobiGr	9c12dc5609	[SoundCloud] Fix SoundCloud ID extraction ... resolveIdWithEmbedPlayer() does not work anymore because the JSON data has been extracted to an API call. For this reason, replace resolveIdWithEmbedPlayer() with resolveIdWithWidgetApi)( which performs the API call.	2021-04-26 17:58:34 +02:00
Fynn Godau	9dc7a1d5ee	[Bandcamp] Add comments (first page)	2021-04-26 17:25:04 +02:00
TobiGr	8f023c1ec7	[SoundCloud] Update client id	2021-04-26 15:14:11 +02:00
Tobi	7f202db8b1	Fix Google URL detection (#617)	2021-04-26 11:49:47 +02:00
fynngodau	6db4bea8ca	Rework Bandcamp comments (#613)	2021-04-13 21:10:59 +02:00
Fynn Godau	3671876721	Fix bandcamp radio stream extractor	2021-04-13 19:30:11 +02:00
Fynn Godau	90b5c00599	Extract related items #593	2021-04-13 17:56:19 +02:00
TobiGr	1fe645704f	[YouTube] FIx testMoreRelatedItems in RandomQueryNoMorePages	2021-04-11 23:21:35 +02:00
TobiGr	9a2b814d8f	Update mocks	2021-04-11 22:59:01 +02:00
TobiGr	50005ce937	[YouTube] Replace unavailable YouTube Stream Segments test ... The stream is unavailable in many countries.	2021-04-11 22:58:48 +02:00
TobiGr	318cec7625	[Bandcamp] Add Radio stream segments	2021-04-11 21:19:44 +02:00
TobiGr	55e0f8e725	Merge branch 'master' into dev	2021-04-10 11:33:34 +02:00
TobiGr	33173eb3e6	Make CONSENT COOKIE constants private	2021-04-09 11:59:17 +02:00
TobiGr	b94b316558	Update mocks	2021-04-09 11:59:17 +02:00
TobiGr	4b8d4a84b6	Set the NumberGenerator for th remaining tests	2021-04-09 11:59:17 +02:00
XiangRongLin	db7ad9c27d	Update Mocks	2021-04-08 16:37:29 +02:00
XiangRongLin	4833df60e5	Add method to inject Random into YoutubeParsingHelper and use in tests	2021-04-08 16:36:55 +02:00
TobiGr	682ec27737	Update mocks	2021-04-08 14:22:31 +02:00
TobiGr	080d1e318d	Code improvement	2021-04-07 14:11:47 +02:00
TobiGr	4ee270fe01	Fix LiveStream Test	2021-04-07 14:11:41 +02:00
TobiGr	883f16e0ad	[YouTube] Set CONSENT cookie	2021-04-07 14:11:37 +02:00
XiangRongLin	aee3838ed5	Ignore unreliable SoundcloudGoPlusTrack.testRelatedItems	2021-04-06 18:26:45 +02:00
XiangRongLin	7b06c696e2	Use Page.body for YoutubeChannelExtractor	2021-04-06 18:14:38 +02:00
XiangRongLin	eda1b6e199	Use Page.body for YoutubePlaylistExtractor	2021-04-06 18:14:38 +02:00
XiangRongLin	3af26a2821	Add body field to Page	2021-04-04 11:39:58 +02:00
XiangRongLin	7b36469807	Fix SoundcloudGoPlusTrack ... https://soundcloud.com/martinsolveig/places#t=0 has related streams	2021-04-04 09:39:26 +02:00
XiangRongLin	3d70b6d46a	Fix a test and ignore rest	2021-04-03 11:03:10 +02:00
XiangRongLin	1925dcf4dc	Merge pull request #597 from fynngodau/related-items ... Rename getRelatedStreams to getRelatedItems and change return type	2021-04-02 19:54:22 +02:00
Fynn Godau	c877712647	Add deprecated old method calls to avoid breaking API	2021-04-01 22:32:36 +02:00
Fynn Godau	14f6f1b7c3	Generify related streams calls and rename method	2021-03-31 20:24:40 +02:00
TobiGr	1498e1905e	Reduce number of type conversions in YouTubeSearchExtractor.collectStreamsFrom by introducing new variable	2021-03-31 13:30:38 +02:00
Tobi	7ea2cd73d3	Merge pull request #589 from fynngodau/bandcamp-more-recent-pages ... Load more featured pages and fix featured cover arts not loading	2021-03-31 12:15:12 +02:00
Fynn Godau	705f6c6e33	Apply review	2021-03-31 11:02:30 +02:00
TobiGr	657b00ca11	Smaller code improvements	2021-03-30 10:10:53 +02:00
TobiGr	8fbd1a2bcf	FIx potential NullPointerException	2021-03-30 10:07:20 +02:00
TobiGr	f2e7c7ebd6	Replace Vector with ArrayList for performance improvements ... I could not figure out, why these two methods are necessary.	2021-03-30 10:07:20 +02:00
TobiGr	48fd8ca7b3	Remove redundant type declaration	2021-03-30 10:07:20 +02:00
TobiGr	22bd16fca3	Make vars private, final and static	2021-03-30 10:07:20 +02:00
TobiGr	6a0f6e846a	Fix possible bug when language.id field is not present and this the languageCode an empty String.	2021-03-30 10:07:20 +02:00
TobiGr	b9282bbe94	Remove checks which are always true	2021-03-30 10:07:20 +02:00
TobiGr	46eab1ec17	Remove declarations of exceptions which are not thrown from method signatures	2021-03-30 10:07:20 +02:00
TobiGr	c3e23559d7	Remove unused imports	2021-03-30 10:07:20 +02:00
TobiGr	e4c40ae6d1	Use List.isEmpty()	2021-03-30 10:07:20 +02:00
Fynn Godau	d663be5a78	[Bandcamp] Fix featured albums not showing thumbnails	2021-03-28 13:12:22 +02:00
Fynn Godau	dbcf61c6f7	[Bandcamp] Load more featured pages	2021-03-28 12:53:05 +02:00
Stypox	b4dee6d08f	Merge pull request #306 from B0pol/metadata ... Extract metadata for youtube, soundcloud & mediaccc	2021-03-27 08:45:47 +01:00
TobiGr	070a40e181	[YouTube] Mix Playlist - Fix getting name	2021-03-25 21:47:16 +01:00
TobiGr	b029779217	[media.cc.de] Disable unreliable tests for now	2021-03-25 19:44:44 +01:00
TobiGr	bbee15474d	[PeerTube] Fix playlist extraction	2021-03-25 19:19:19 +01:00
TiA4f8R	ae283314da	Add a comment explaining subscribers count calculation	2021-03-25 17:54:46 +01:00
TiA4f8R	abb790f465	Remove println in getSubscriberCount from PeertubeAccountExtractor class	2021-03-25 17:49:22 +01:00
TiA4f8R	772de53a66	Increase the number of subscribers in PeertubeAccountExtractorTest	2021-03-25 17:49:20 +01:00
TiA4f8R	efe2b964f0	Extract real subscribers of a Peertube account ... Apply fix provided in the PR and move all accounts/ strings to a constant, ACCOUNTS.	2021-03-25 17:47:44 +01:00
Tobi	12835bfae1	Merge pull request #551 from FireMasterK/ff-privacy ... Change UA to privacy.resistFingerprinting.	2021-03-24 17:58:34 +01:00
Tobi	cae6cea551	Merge pull request #582 from Stypox/fix-trending ... [YouTube] Fix trending getName()	2021-03-24 10:49:46 +01:00
Stypox	ce7cbbc9a0	[YouTube] Fix trending getName()	2021-03-24 09:14:09 +01:00
XiangRongLin	9009f1e277	Merge pull request #517 from XiangRongLin/flaky_test ... Add MockOnlyRule to allow skipping specific tests based on downloader	2021-03-20 18:04:14 +01:00
bopol	152221c7fb	ignore a test with encoding problems	2021-03-19 10:14:43 +01:00
bopol	8806fb4e6b	address reviews	2021-03-19 10:14:43 +01:00
bopol	c47cc54908	Extract metadata for YouTube, SoundCloud & MediaCCC	2021-03-19 10:14:43 +01:00
TiA4f8R	379d7312fa	Don't use a regular expression to find the last segment URL and do code improvements ... Apply suggestions provided in the PR and remove a redundant import.	2021-03-14 17:54:22 +01:00
TiA4f8R	0e3e420a25	Fix tests	2021-03-14 17:54:21 +01:00
TiA4f8R	d61d9d116d	Refactor getAudioStreams method of SoundcloudStreamExtractor ... Split the method into private methods, in order to have a better reading.	2021-03-14 17:54:20 +01:00
TiA4f8R	a7b15b51e6	Change t to transcodingJsonObject in SoundcloudStreamExtractor	2021-03-14 17:54:19 +01:00
TiA4f8R	0438828e36	Add a test for the number of audioStreams in CreativeCommonsPlaysWellWithOthers test ... It should be only two audio streams for track "Plays Well with Others, Ep 2: What Do an Army of Ants and an Online Encyclopedia Have in Common?" by Creative Commons (https://soundcloud.com/wearecc/plays-well-with-others-ep-2-what-do-an-army-of-ants-and-an-online-encyclopedia-have-in-common): - one which is a progressive stream, in MP3 format with a bitrate of 128 kbps - one which is an HLS stream, in OPUS format with a bitrate of 64 kbps.	2021-03-14 17:54:18 +01:00
TiA4f8R	3bd08a2880	Adress requested changes and use final where possible in SoundcloudStreamExtractor ... This commit moved the HLS parsing task to a separate method, did little performance improvements and used final where possible in the SoundcloudStreamExtractor file.	2021-03-14 17:54:17 +01:00
TiA4f8R	cbacd3c0a5	Add a check to don't show MP3 128kbps stream twice and catch IOException when fetching the HLS Manifest ... If a progressive stream is present in the transcodings, it's unnecessary to show twice an MP3 128kbps stream so if this is the case, the MP3 HLS stream will be not added to the audioStreams, else it will. This commit also catch fetching errors in HLS manifests parsing and don't add the corresponding stream if an error occurs.	2021-03-14 17:54:16 +01:00
TiA4f8R	26f1b4e7dc	Support SoundCloud HLS by using a workaround ... This commit tries to support SoundCloud HLS streams by parsing M3U manifests, get the last segment URL (in order to get track length) and request a segment URL equals to track's duration so it's a single URL.	2021-03-14 17:54:14 +01:00
Fynn Godau	2e57a8f24f	[Bandcamp] Fix link handler acceptance behaviour ... * Test for bandcamp footer instead of meta tag (which is not present on all pages) * Accept links to /music, not just /releases * Correctly handle uppercase URLs	2021-03-14 09:51:30 +01:00
TobiGr	70d9e389b9	[Bandcamp] Add tests for external bandcamp artists ... Necessary, because the external pages tested before were converted to bandcamp.com pages. See f9d06252f2	2021-03-13 20:51:59 +01:00
TobiGr	f9d06252f2	[Bandcamp] Fix tests	2021-03-13 17:42:39 +01:00
Tobi	7e6f464407	Merge pull request #509 from TiA4f8R/soundcloud-improvements ... Add new exceptions to be able to display different error messages in apps	2021-03-07 13:51:15 +01:00
Tobi	ac51134aed	Merge pull request #546 from XiangRongLin/unignore2 ... Unignore more tests	2021-03-07 13:15:24 +01:00
TobiGr	22fa131922	Merge branch 'dev' into bandcamp	2021-03-05 19:45:37 +01:00
TiA4f8R	e55284bb8f	Code style improvements ... Use final where possible in YoutubeStreamExtractor and do some other code style improvements	2021-03-05 16:38:53 +01:00
TobiGr	d0a1041afd	Prevent NullPointerException and use equalsIgnorecase	2021-03-05 16:38:52 +01:00
TiA4f8R	890cbba625	Update PaidContent exception	2021-03-05 16:38:51 +01:00
TobiGr	da3cfa967d	Handle age-restricted videos	2021-03-05 16:38:50 +01:00
TiA4f8R	448b68700c	Use final in exceptions and add AgeRestrictedException ... AgeRestrictedException will be thrown only if the reason message equals to "Sign in to confirm your age" and if the age limit is 18.	2021-03-05 16:38:49 +01:00
TobiGr	22e6f33f0a	Ignore SoundCloudGeoRestrictedTest until HLS stream extraction is added ... The ContentNotSupportedException is thrown because no supported audio streams where extracted. However, SoundCLoud does not check, whether there are any streams available. This commit should be reverted in #526	2021-03-05 16:38:47 +01:00
TobiGr	21158744b1	Add expectedDescriptionIsEmpty() to DefaultStreamExtractorTest ... Also check if related streams are empty if they are expected to be empty.	2021-03-05 16:38:46 +01:00
TiA4f8R	f15d7837a1	Fix SoundCloud tests	2021-03-05 16:38:45 +01:00
TiA4f8R	4e45aef2b3	Fix SoundCloud CreativeCommonsPlaysWellWithOthers test	2021-03-05 16:38:43 +01:00
TiA4f8R	59d6d3f04e	Fix YouTube tests	2021-03-05 16:38:42 +01:00
TiA4f8R	3925204658	Add tests and rename SoundcloudGoPlusException to SoundcloudGoPlusContentException	2021-03-05 16:38:40 +01:00
TiA4f8R	771bb1a2cb	Add three new exceptions for unavailable YouTube contents ... These exceptions are thrown on a test with the error messages text, because YouTube returns only "UNPLAYABLE" status in most error cases. Tests are based with English strings, so changing the lang used by extractor will throw the generic exception (ContentNotAvailableException).	2021-03-05 16:38:39 +01:00
TiA4f8R	35325d980d	Add GeographicRestrictionException and SoundCloudGoPlusException in NewPipe Extractor to be able to display different error messages ... This commit adds two new exceptions in NewPipe Extractor: GeographicRestrictionException and SoundCloudGoPlusException (which extend to ContentNotAvailableException). These exceptions allow showing different error messages to user when a content isn't available in his/her/its country (only used for now by SoundCloudStreamExtractor) or when the content is a SoundCloud Go+ track.	2021-03-05 16:38:37 +01:00
Tobi	bc0cda68d1	Merge pull request #570 from TiA4f8R/fix-yt-channels-playlists ... Fix YouTube channels contination	2021-03-05 15:26:49 +01:00
Tobi	a3c6fceef5	Merge pull request #573 from B0pol/comments-performance ... [youtube] improve comments extraction performance	2021-03-05 14:46:27 +01:00
bopol	ff5273b882	Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeCommentsExtractor.java ... Co-authored-by: Tobi <TobiGr@users.noreply.github.com>	2021-03-05 14:39:01 +01:00
Tobi	ec1127dd6a	Merge pull request #571 from TeamNewPipe/soundcloud_client_id ... [SoundCloud] Update client_id and fix extraction	2021-03-05 14:21:14 +01:00
bopol	ed850d0688	[youtube] improve comments extraction performance ... - do not parse responseBody twice for continuation instead try to get commentsTokenInside with the new pattern ("sectionListRenderer") and try again with the old pattern ("commentSectionRenderer") on failure - do not unescape responseBody multiple times -> parse responseBody less times	2021-03-05 13:37:08 +01:00
TobiGr	440a808b8a	Update mocks for channels	2021-03-05 13:13:36 +01:00
golfinq	05213175cd	minor edit, made variable final	2021-03-05 02:49:15 -05:00
golfinq	8bceb57a6c	formatting fix	2021-03-05 02:46:06 -05:00
golfinq	ae48bdea4c	Updated to reflect variable name change	2021-03-05 02:40:36 -05:00
TiA4f8R	b27efdc9a4	Remove clickTrackingParams in getNextPage method of YoutubeChannelExtractor ... Because it's unused with the new request. Co-authored-by: bopol <bopol@e.email>	2021-03-04 22:50:20 +01:00
TobiGr	2b671b15ce	[YouTube] Use new continuation API for channels ... Co-authored-by: TiA4f8R <74829229+tia4f8r@users.noreply.github.com>	2021-03-04 21:47:16 +01:00
TobiGr	33594d4aff	[SoundCloud] Fix client id extraction ... SoundCloud changed their Javascript structure. That caused the client_id to be in the middle of the file (at byte ~43000 ). To have a little buffer, we now fetch the first 50000 bytes.	2021-03-04 21:32:34 +01:00
TobiGr	fb2f37d223	[SoundCloud] Update hardcoded client id	2021-03-04 21:25:41 +01:00
XiangRongLin	506cc5fbe9	regenerate mocks	2021-03-04 19:01:58 +01:00
XiangRongLin	03b00ff1d6	Move toJsonArray and toJsonObject to JsonUtils	2021-03-04 18:58:51 +01:00
FireMasterK	668b080b3a	Use dynamic key.	2021-03-04 08:45:14 +01:00
XiangRongLin	43b46bd408	Use new youtube continuations api for playlists ... Requires sending a POST request instead of GET. clientName and clientVersion, which were required as headers previously now need to be part of the request payload. continuation id also needs to be part of request body. quick and dirty solution.	2021-03-03 19:49:26 +01:00
XiangRongLin	beb05bd05c	Replace test url, because playlist was deleted	2021-03-03 19:46:14 +01:00
XiangRongLin	e13e237392	Fix typo and reword some explanations	2021-02-26 17:48:03 +01:00
TobiGr	a1688fe953	Move BandcampExtractorHelper.getJsonData(String, String) to JsonUtils	2021-02-21 13:51:12 +01:00
TobiGr	70814dcfef	Fix Utils.nonEmptyAndNullJoin ... When using the index here, it the index needs to be decremented once an element is removed. To cirecumvent this, the native Collections.removeIf() method is used.	2021-02-21 12:51:47 +01:00
TobiGr	b9e8ee8450	Rename BandcampExtractorHelper.smartConcatenate(String[], String) to Utils.nonEmptyAndNullJoin(String, String[])	2021-02-21 12:35:14 +01:00
TobiGr	c07db80ef0	Add BASE_URL and BASE_API_URL to BandcampExtractorHelper	2021-02-21 12:15:45 +01:00
TobiGr	adde4332d1	Code improvements	2021-02-20 23:58:54 +01:00
FireMasterK	42bdd8adc5	Update to fix failing mock tests.	2021-02-20 17:14:41 +05:30
FireMasterK	e8bccfaf5d	Change UA to privacy.resistFingerprinting.	2021-02-20 17:14:41 +05:30
bopol	0c831afa7b	use DownloaderFactory instead of DownloaderTestImpl for search ChannelVerified	2021-02-20 11:41:30 +01:00
TobiGr	ea120a4637	remove print stacktrace ... use interface	2021-02-20 00:23:45 +01:00
XiangRongLin	adf9d7d10f	Add reason field to MockOnly ... This enforces developers to document why a test is skipped	2021-02-19 18:54:18 +01:00
XiangRongLin	ea52030613	Add MockOnlyRule to allow skipping specific tests based on downloader	2021-02-19 18:54:17 +01:00
TobiGr	02920fafa8	Add isUploaderVerified()	2021-02-19 15:51:23 +01:00
TobiGr	91e9309486	Merge remote-tracking branch 'origin/dev' into bandcamp	2021-02-19 13:37:45 +01:00
TobiGr	fa61b864f2	Code improvements	2021-02-19 13:37:24 +01:00
TobiGr	98268e351c	Move radio URL check into a function	2021-02-19 13:35:44 +01:00
FireMasterK	5333d8a98b	Add more data.	2021-02-19 00:55:03 +05:30
bopol	1a322ad8ed	Add uploader verified by service extraction	2021-02-18 16:36:40 +01:00
FireMasterK	c24afa2cbb	Permalink URL.	2021-02-18 20:16:53 +05:30
FireMasterK	8d54401233	Add fps field as well.	2021-02-18 20:16:53 +05:30
FireMasterK	5b70645d85	Add data to muxed streams aswell.	2021-02-18 20:16:53 +05:30
FireMasterK	e975d33fbe	Change jdoc / comments.	2021-02-18 20:16:53 +05:30
FireMasterK	f31b2a68fd	use getters/setters.	2021-02-18 20:16:53 +05:30
FireMasterK	d4945ac55c	Constructor cleanup.	2021-02-18 20:16:52 +05:30
FireMasterK	3f6a601be8	Fix jdoc / comments.	2021-02-18 20:16:52 +05:30
FireMasterK	a7c9905183	Fix requested changes.	2021-02-18 20:16:52 +05:30
FireMasterK	0c0f2d74bc	Make fields private ... Co-authored-by: bopol <bopol@e.email>	2021-02-18 20:16:52 +05:30
FireMasterK	d1054338d3	Fix error with livestreams.	2021-02-18 20:16:52 +05:30
FireMasterK	11eb4932f4	Add data to respective classes.	2021-02-18 20:16:52 +05:30
FireMasterK	525e345ed8	remove duplicate field.	2021-02-18 20:16:52 +05:30
FireMasterK	e844d2aed3	Add dash fields to the ItagItem.	2021-02-18 20:16:52 +05:30
bopol	64f6b0478f	Merge pull request #545 from XiangRongLin/comment_reset ... Add comment explaining YoutubeParsingHelper.resetClientVersionAndKey	2021-02-18 15:14:03 +01:00
XiangRongLin	a64579da2f	Unignore some tests that work on Linux	2021-02-17 20:31:26 +01:00
XiangRongLin	b99cb8624f	Unignore PublicBroadcasterTest ... Tests only work on linux, mocks have to be generated on linux too	2021-02-17 20:31:26 +01:00
XiangRongLin	37366c65ca	Remove encoding of '&' to '&amp;' in test and unignore it	2021-02-17 20:31:26 +01:00
XiangRongLin	c327922cc0	Add explanations to broken tests	2021-02-17 20:31:26 +01:00
XiangRongLin	26d50aca05	Add comment explaining YoutubeParsingHelper.resetClientVersionAndKey	2021-02-17 19:21:39 +01:00
TobiGr	54aa5b3042	Use propper structure in KioskExtractors ... Made BandCampRadioExtractor a Kiosk which holds StreamInfoItems and not InfoItems.	2021-02-16 21:09:42 +01:00
TobiGr	54b8e54f80	Fix potential NPE	2021-02-16 20:59:13 +01:00
TobiGr	16973126a0	Use Collections.emptyList() instead of Collections.EMPTY_LIST or new ArrayList()	2021-02-16 20:25:48 +01:00
bopol	84380e4c3a	Merge pull request #536 from B0pol/watchendpoints ... support /watch/, /v/ and /w/ subpaths	2021-02-16 20:23:27 +01:00
bopol	41f689b099	Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/linkHandler/YoutubeStreamLinkHandlerFactory.java ... Co-authored-by: Stypox <stypox@pm.me>	2021-02-16 19:17:10 +01:00
XiangRongLin	a9594f531f	Merge pull request #543 from XiangRongLin/unignore1 ... Unignore and fix some tests	2021-02-16 18:00:16 +01:00
XiangRongLin	a28e0226c5	Merge pull request #519 from XiangRongLin/better_logging_error_handling ... Better logging, error handling, doc for mock tests	2021-02-16 15:06:06 +01:00
XiangRongLin	05cb22c654	Unignore Gronkh.testDescription ... Remove special character ü from assertion	2021-02-16 08:59:35 +01:00
XiangRongLin	d9b4c2779f	Unignore YoutubeSearchQHTest.testRegularValues	2021-02-16 08:59:35 +01:00
XiangRongLin	137272b7a4	Unignore YoutubeCommentsExtractorTest.EmptyComment ... Mock files were missing	2021-02-16 08:59:34 +01:00
TobiGr	5bf9fddba9	Code improvements	2021-02-15 22:45:17 +01:00
TobiGr	e062c8cb0d	Merge branch 'dev' into bandcamp	2021-02-15 22:09:41 +01:00
bopol	7a3d9bdb7d	Description: add .equals() and .hashCode()	2021-02-15 18:51:47 +01:00
bopol	557934cb17	use EMPTY_STRING	2021-02-15 18:51:46 +01:00
bopol	9ca52ca68a	use only one constant UTF-8	2021-02-15 18:47:54 +01:00
XiangRongLin	50e5718bd1	Merge pull request #518 from XiangRongLin/generate_mocks ... Use and generate mocks for youtube tests	2021-02-15 06:39:41 +01:00
Tobi	d7486f251a	Merge pull request #538 from B0pol/soundcloud-georestriction ... Soundcloud: fix charts extraction when georestricted	2021-02-14 23:52:27 +01:00
TobiGr	432c68cdec	Update okhttp from 3.12.11 to 3.12.13 ... Changelog https://square.github.io/okhttp/changelog_3x/#version-31213	2021-02-14 23:44:19 +01:00
XiangRongLin	a88b2e6af4	Undo formatting of YoutubeStreamExtractorDefaultTest	2021-02-14 14:57:23 +01:00
bopol	8d89c82caa	Soundcloud: fix charts extraction when georestricted	2021-02-13 22:28:31 +01:00
bopol	ee2012c948	rename folder to subpath	2021-02-13 22:25:46 +01:00
bopol	8db40db4de	use substring instead of split	2021-02-13 22:20:05 +01:00
bopol	81317cf6bc	youtube: support /w/ folder	2021-02-13 22:20:05 +01:00
bopol	f15c0fcfed	support /watch/ and /v/ folders	2021-02-13 22:20:05 +01:00
XiangRongLin	971128c464	Ignore failing tests	2021-02-13 20:28:03 +01:00
XiangRongLin	92dea0806a	Regenerate json files for youtube tests	2021-02-13 20:28:03 +01:00
XiangRongLin	d6b9930c18	Use DownloaderFactory in YoutubeCommentsExtractorTest.Pinned	2021-02-13 20:12:39 +01:00
XiangRongLin	b555f38fa5	Use real downloader for YoutubeSubscriptionExtractorTest ... It doesn't make network requests anyway	2021-02-13 20:12:39 +01:00
XiangRongLin	f352b0c7af	YoutubeMusicSearchExtractorTest can't be used with mocks ... Requires further research	2021-02-13 20:00:51 +01:00
XiangRongLin	f45b5610bf	Unignore YoutubeChannelLocalizationTest ... With mocks time is not a concern anymore	2021-02-13 20:00:51 +01:00
XiangRongLin	19737e06ee	Use DownloaderFactory in YoutubeChannelExtractorTest ... KurzgesagtAdditional needed to be extracted because it ran a duplicated request.	2021-02-13 20:00:51 +01:00
XiangRongLin	84eb4b30c9	Use DownloaderFactory in YouTube tests	2021-02-13 20:00:51 +01:00
XiangRongLin	b43f3474f5	Make resetClientVersionAndKey public	2021-02-13 20:00:49 +01:00
bopol	2859c7ba5c	add comment explaining why .substring(23)	2021-02-13 12:10:41 +01:00
bopol	73353996d7	fix more tests	2021-02-12 23:18:38 +01:00
bopol	a931e31239	fix tests	2021-02-12 22:36:18 +01:00
TobiGr	d116680ea3	Merge branch 'master' into dev	2021-02-12 22:26:35 +01:00
bopol	b0f356dd76	YT channels: fix continuation	2021-02-12 14:39:09 +01:00
bopol	fcdb50b825	make getInfo from Extractor public ... In StreamInfo and CommentsInfo	2021-02-09 14:58:43 +01:00
bopol	44c54d403a	fix Invidious channels being accepted by YoutubeStreamLinkHandlerFactory (#527) ... fixes https://github.com/TeamNewPipe/NewPipeExtractor/issues/524, see it for more details	2021-02-05 18:57:58 +01:00
TiA4f8R	1414a6f178	Fix extraction of YouTube chapters (#531) ... YouTube changed the name of the chapters in the JSON from engagement-panel-macro-markers to engagement-panel-macro-markers-description-chapters, so extracting chapters doesn't work.	2021-02-03 22:07:34 +01:00
Tobias Groza	bfa639950e	Merge pull request #528 from Stypox/ytmusic-0 ... [YouTube Music] Fix extracting search item view/subscriber count when = 0	2021-01-29 23:23:23 +01:00
TobiGr	d728c4fad0	Remove unused imports ... I searched for "StandardCharsets" to prevent possible bugs with Android KitKat. The import is unused and is therefor removed. There are other usages of the StandardCharsets class in the test implementation, but those are not relevant.	2021-01-29 23:16:40 +01:00
Stypox	a64dfd7343	[YouTube Music] Fix extracting search item view/subscriber count when = 0	2021-01-29 16:44:24 +01:00
bopol	2662ceca22	rename getHeartedByCreator and getPinned to isPinned and isHeartedByCreator	2021-01-23 13:06:07 +01:00
bopol	66e4eb2f96	extract whether the comment is pinned	2021-01-22 23:59:22 +01:00
bopol	67ddfefdc6	keep invidious instances list in the same order ... as suggested by @TobiGr	2021-01-22 20:58:14 +01:00
bopol	53d3f7989b	update invidious instances list	2021-01-22 19:30:25 +01:00
XiangRongLin	88e4c8667b	Add comments to RecordingDownloader about usage	2021-01-17 19:53:20 +01:00