TeamPiped-mirror/Piped-Backend
1
0
Fork 0
mirror of https://github.com/TeamPiped/Piped-Backend.git synced 2024-08-14 23:51:41 +00:00
Code Issues Projects Releases Packages Wiki Activity
Piped-Backend/src/main/java/me/kavin/piped/utils/ResponseHelper.java

1166 lines
49 KiB
Java
Raw Normal View History

Initial commit.
2020-11-12 21:19:45 +00:00
package me.kavin.piped.utils;
Improve livestream detection.
2021-10-01 17:53:54 +00:00
import static me.kavin.piped.consts.Constants.YOUTUBE_SERVICE;
Initial commit.
2020-11-12 21:19:45 +00:00
import java.io.IOException;
Log pubsub failures.
2021-07-22 09:43:27 +00:00
import java.io.InputStream;
Initial commit.
2020-11-12 21:19:45 +00:00
import java.net.MalformedURLException;
import java.net.URI;
Check if passwords are compromised in HIBP.
2021-09-05 19:19:54 +00:00
import java.net.URISyntaxException;
Initial commit.
2020-11-12 21:19:45 +00:00
import java.net.URL;
import java.net.http.HttpRequest;
import java.net.http.HttpRequest.BodyPublishers;
Use Hibernate as a database (#32) * Basic hibernate structure. * Commit stash. * Improve a lot of things. * Implement Feed, PubSub, query optimizations. * Update video views on channel visits. * Allow mass importing subscriptions. * Allow configuring database in config.properties. * Major multi-threading and pubsub improvements. * PubSub query improvements and fix for postgres. * Revert docker-compose change. * Disable showing sql statements.
2021-07-16 22:40:46 +00:00
import java.net.http.HttpRequest.Builder;
Log pubsub failures.
2021-07-22 09:43:27 +00:00
import java.net.http.HttpResponse;
Initial commit.
2020-11-12 21:19:45 +00:00
import java.net.http.HttpResponse.BodyHandlers;
Add Playlist RSS (#80) * WIP Playlist rss. * Complete adding support.
2021-07-04 18:57:18 +00:00
import java.nio.charset.StandardCharsets;
Use Hibernate as a database (#32) * Basic hibernate structure. * Commit stash. * Improve a lot of things. * Implement Feed, PubSub, query optimizations. * Update video views on channel visits. * Allow mass importing subscriptions. * Allow configuring database in config.properties. * Major multi-threading and pubsub improvements. * PubSub query improvements and fix for postgres. * Revert docker-compose change. * Disable showing sql statements.
2021-07-16 22:40:46 +00:00
import java.security.NoSuchAlgorithmException;
import java.security.spec.InvalidKeySpecException;
Commit missing import and fix build.
2021-08-28 08:06:09 +00:00
import java.util.Arrays;
Add support for search filters and simplify continuations.
2021-06-14 19:44:23 +00:00
import java.util.Collections;
Generate RSS feed for subscriptions.
2021-07-19 19:44:18 +00:00
import java.util.Date;
Initial commit.
2020-11-12 21:19:45 +00:00
import java.util.List;
Use Hibernate as a database (#32) * Basic hibernate structure. * Commit stash. * Improve a lot of things. * Implement Feed, PubSub, query optimizations. * Update video views on channel visits. * Allow mass importing subscriptions. * Allow configuring database in config.properties. * Major multi-threading and pubsub improvements. * PubSub query improvements and fix for postgres. * Revert docker-compose change. * Disable showing sql statements.
2021-07-16 22:40:46 +00:00
import java.util.Map;
Livestream and Async task for fetching data.
2020-11-17 05:34:50 +00:00
import java.util.concurrent.CompletableFuture;
Add lbryId to streams response.
2021-08-08 11:17:58 +00:00
import java.util.concurrent.ExecutionException;
Initial commit.
2020-11-12 21:19:45 +00:00
import java.util.concurrent.TimeUnit;
Use Hibernate as a database (#32) * Basic hibernate structure. * Commit stash. * Improve a lot of things. * Implement Feed, PubSub, query optimizations. * Update video views on channel visits. * Allow mass importing subscriptions. * Allow configuring database in config.properties. * Major multi-threading and pubsub improvements. * PubSub query improvements and fix for postgres. * Revert docker-compose change. * Disable showing sql statements.
2021-07-16 22:40:46 +00:00
import javax.persistence.criteria.CriteriaBuilder;
import javax.persistence.criteria.CriteriaQuery;
import javax.persistence.criteria.Root;
Check if passwords are compromised in HIBP.
2021-09-05 19:19:54 +00:00
import org.apache.commons.codec.digest.DigestUtils;
Log pubsub failures.
2021-07-22 09:43:27 +00:00
import org.apache.commons.io.IOUtils;
Use substringAfter to cleanup code and fix YouTube music URLs.
2021-09-05 18:44:24 +00:00
import org.apache.commons.lang3.StringUtils;
Livestream and Async task for fetching data.
2020-11-17 05:34:50 +00:00
import org.apache.commons.lang3.exception.ExceptionUtils;
Use Hibernate as a database (#32) * Basic hibernate structure. * Commit stash. * Improve a lot of things. * Implement Feed, PubSub, query optimizations. * Update video views on channel visits. * Allow mass importing subscriptions. * Allow configuring database in config.properties. * Major multi-threading and pubsub improvements. * PubSub query improvements and fix for postgres. * Revert docker-compose change. * Disable showing sql statements.
2021-07-16 22:40:46 +00:00
import org.hibernate.Session;
Initial commit.
2020-11-12 21:19:45 +00:00
import org.json.JSONObject;
Next page in search results.
2020-12-14 07:11:42 +00:00
import org.schabi.newpipe.extractor.InfoItem;
Channel Pages and search suggestions.
2020-11-25 05:26:25 +00:00
import org.schabi.newpipe.extractor.ListExtractor.InfoItemsPage;
import org.schabi.newpipe.extractor.Page;
Initial commit.
2020-11-12 21:19:45 +00:00
import org.schabi.newpipe.extractor.channel.ChannelInfo;
Add additional details and playlists on search endpoint.
2021-05-28 12:08:32 +00:00
import org.schabi.newpipe.extractor.channel.ChannelInfoItem;
Initial commit.
2020-11-12 21:19:45 +00:00
import org.schabi.newpipe.extractor.comments.CommentsInfo;
Add comments API.
2021-04-03 15:08:32 +00:00
import org.schabi.newpipe.extractor.comments.CommentsInfoItem;
Initial commit.
2020-11-12 21:19:45 +00:00
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
import org.schabi.newpipe.extractor.exceptions.ParsingException;
Support region localization for trending (#79) * Support region localization for trending * Revert linting, remove defaulting to US, and fix warning. Co-authored-by: FireMasterK <20838718+FireMasterK@users.noreply.github.com>
2021-07-04 17:53:33 +00:00
import org.schabi.newpipe.extractor.kiosk.KioskExtractor;
Initial commit.
2020-11-12 21:19:45 +00:00
import org.schabi.newpipe.extractor.kiosk.KioskInfo;
Support region localization for trending (#79) * Support region localization for trending * Revert linting, remove defaulting to US, and fix warning. Co-authored-by: FireMasterK <20838718+FireMasterK@users.noreply.github.com>
2021-07-04 17:53:33 +00:00
import org.schabi.newpipe.extractor.kiosk.KioskList;
import org.schabi.newpipe.extractor.localization.ContentCountry;
add playlists and small changes.
2021-01-04 05:47:28 +00:00
import org.schabi.newpipe.extractor.playlist.PlaylistInfo;
Add additional details and playlists on search endpoint.
2021-05-28 12:08:32 +00:00
import org.schabi.newpipe.extractor.playlist.PlaylistInfoItem;
Add search.
2020-12-09 13:30:42 +00:00
import org.schabi.newpipe.extractor.search.SearchInfo;
Initial commit.
2020-11-12 21:19:45 +00:00
import org.schabi.newpipe.extractor.stream.StreamInfo;
import org.schabi.newpipe.extractor.stream.StreamInfoItem;
Improve livestream detection.
2021-10-01 17:53:54 +00:00
import org.schabi.newpipe.extractor.stream.StreamType;
Use Hibernate as a database (#32) * Basic hibernate structure. * Commit stash. * Improve a lot of things. * Implement Feed, PubSub, query optimizations. * Update video views on channel visits. * Allow mass importing subscriptions. * Allow configuring database in config.properties. * Major multi-threading and pubsub improvements. * PubSub query improvements and fix for postgres. * Revert docker-compose change. * Disable showing sql statements.
2021-07-16 22:40:46 +00:00
import org.springframework.security.crypto.argon2.Argon2PasswordEncoder;
Add support for legacy bcrypt hashes from Invidious
2021-10-03 08:02:01 +00:00
import org.springframework.security.crypto.bcrypt.BCryptPasswordEncoder;
Initial commit.
2020-11-12 21:19:45 +00:00
Channel Pages and search suggestions.
2020-11-25 05:26:25 +00:00
import com.fasterxml.jackson.core.JsonProcessingException;
Initial commit.
2020-11-12 21:19:45 +00:00
import com.github.benmanes.caffeine.cache.Caffeine;
import com.github.benmanes.caffeine.cache.LoadingCache;
Add Playlist RSS (#80) * WIP Playlist rss. * Complete adding support.
2021-07-04 18:57:18 +00:00
import com.rometools.rome.feed.synd.SyndEntry;
import com.rometools.rome.feed.synd.SyndEntryImpl;
import com.rometools.rome.feed.synd.SyndFeed;
import com.rometools.rome.feed.synd.SyndFeedImpl;
Generate RSS feed for subscriptions.
2021-07-19 19:44:18 +00:00
import com.rometools.rome.feed.synd.SyndPerson;
import com.rometools.rome.feed.synd.SyndPersonImpl;
Add Playlist RSS (#80) * WIP Playlist rss. * Complete adding support.
2021-07-04 18:57:18 +00:00
import com.rometools.rome.io.FeedException;
import com.rometools.rome.io.SyndFeedOutput;
Initial commit.
2020-11-12 21:19:45 +00:00
Use Hibernate as a database (#32) * Basic hibernate structure. * Commit stash. * Improve a lot of things. * Implement Feed, PubSub, query optimizations. * Update video views on channel visits. * Allow mass importing subscriptions. * Allow configuring database in config.properties. * Major multi-threading and pubsub improvements. * PubSub query improvements and fix for postgres. * Revert docker-compose change. * Disable showing sql statements.
2021-07-16 22:40:46 +00:00
import it.unimi.dsi.fastutil.objects.Object2ObjectOpenHashMap;
Initial commit.
2020-11-12 21:19:45 +00:00
import it.unimi.dsi.fastutil.objects.ObjectArrayList;
import me.kavin.piped.consts.Constants;
Commit changes.
2021-02-24 09:52:29 +00:00
import me.kavin.piped.ipfs.IPFS;
Initial commit.
2020-11-12 21:19:45 +00:00
import me.kavin.piped.utils.obj.Channel;
Commit changes.
2021-02-24 09:52:29 +00:00
import me.kavin.piped.utils.obj.ChapterSegment;
Add comments API.
2021-04-03 15:08:32 +00:00
import me.kavin.piped.utils.obj.Comment;
import me.kavin.piped.utils.obj.CommentsPage;
Use Hibernate as a database (#32) * Basic hibernate structure. * Commit stash. * Improve a lot of things. * Implement Feed, PubSub, query optimizations. * Update video views on channel visits. * Allow mass importing subscriptions. * Allow configuring database in config.properties. * Major multi-threading and pubsub improvements. * PubSub query improvements and fix for postgres. * Revert docker-compose change. * Disable showing sql statements.
2021-07-16 22:40:46 +00:00
import me.kavin.piped.utils.obj.FeedItem;
Channel Pages and search suggestions.
2020-11-25 05:26:25 +00:00
import me.kavin.piped.utils.obj.PipedStream;
add playlists and small changes.
2021-01-04 05:47:28 +00:00
import me.kavin.piped.utils.obj.Playlist;
Next page in search results.
2020-12-14 07:11:42 +00:00
import me.kavin.piped.utils.obj.SearchResults;
Initial commit.
2020-11-12 21:19:45 +00:00
import me.kavin.piped.utils.obj.StreamItem;
import me.kavin.piped.utils.obj.Streams;
add playlists and small changes.
2021-01-04 05:47:28 +00:00
import me.kavin.piped.utils.obj.StreamsPage;
Add route to see subscriptions.
2021-07-22 20:04:21 +00:00
import me.kavin.piped.utils.obj.SubscriptionChannel;
Initial commit.
2020-11-12 21:19:45 +00:00
import me.kavin.piped.utils.obj.Subtitle;
Use Hibernate as a database (#32) * Basic hibernate structure. * Commit stash. * Improve a lot of things. * Implement Feed, PubSub, query optimizations. * Update video views on channel visits. * Allow mass importing subscriptions. * Allow configuring database in config.properties. * Major multi-threading and pubsub improvements. * PubSub query improvements and fix for postgres. * Revert docker-compose change. * Disable showing sql statements.
2021-07-16 22:40:46 +00:00
import me.kavin.piped.utils.obj.db.PubSub;
import me.kavin.piped.utils.obj.db.User;
import me.kavin.piped.utils.obj.db.Video;
Add additional details and playlists on search endpoint.
2021-05-28 12:08:32 +00:00
import me.kavin.piped.utils.obj.search.SearchChannel;
import me.kavin.piped.utils.obj.search.SearchPlaylist;
Use Hibernate as a database (#32) * Basic hibernate structure. * Commit stash. * Improve a lot of things. * Implement Feed, PubSub, query optimizations. * Update video views on channel visits. * Allow mass importing subscriptions. * Allow configuring database in config.properties. * Major multi-threading and pubsub improvements. * PubSub query improvements and fix for postgres. * Revert docker-compose change. * Disable showing sql statements.
2021-07-16 22:40:46 +00:00
import me.kavin.piped.utils.resp.AcceptedResponse;
import me.kavin.piped.utils.resp.AlreadyRegisteredResponse;
import me.kavin.piped.utils.resp.AuthenticationFailureResponse;
Check if passwords are compromised in HIBP.
2021-09-05 19:19:54 +00:00
import me.kavin.piped.utils.resp.CompromisedPasswordResponse;
Add configuration property to disable registration.
2021-11-09 06:27:29 +00:00
import me.kavin.piped.utils.resp.DisabledRegistrationResponse;
Use Hibernate as a database (#32) * Basic hibernate structure. * Commit stash. * Improve a lot of things. * Implement Feed, PubSub, query optimizations. * Update video views on channel visits. * Allow mass importing subscriptions. * Allow configuring database in config.properties. * Major multi-threading and pubsub improvements. * PubSub query improvements and fix for postgres. * Revert docker-compose change. * Disable showing sql statements.
2021-07-16 22:40:46 +00:00
import me.kavin.piped.utils.resp.IncorrectCredentialsResponse;
Handle invalid requests gracefully.
2021-07-20 21:03:44 +00:00
import me.kavin.piped.utils.resp.InvalidRequestResponse;
Use Hibernate as a database (#32) * Basic hibernate structure. * Commit stash. * Improve a lot of things. * Implement Feed, PubSub, query optimizations. * Update video views on channel visits. * Allow mass importing subscriptions. * Allow configuring database in config.properties. * Major multi-threading and pubsub improvements. * PubSub query improvements and fix for postgres. * Revert docker-compose change. * Disable showing sql statements.
2021-07-16 22:40:46 +00:00
import me.kavin.piped.utils.resp.LoginResponse;
import me.kavin.piped.utils.resp.SubscribeStatusResponse;
Initial commit.
2020-11-12 21:19:45 +00:00
public class ResponseHelper {
public static final LoadingCache<String, CommentsInfo> commentsCache = Caffeine.newBuilder()
Commit changes.
2021-02-24 09:52:29 +00:00
.expireAfterWrite(1, TimeUnit.HOURS).maximumSize(1000)
.build(key -> CommentsInfo.getInfo("https://www.youtube.com/watch?v=" + key));
Initial commit.
2020-11-12 21:19:45 +00:00
performance: read and write data as byte arrays.
2021-01-12 08:15:09 +00:00
public static final byte[] streamsResponse(String videoId) throws Exception {
Livestream and Async task for fetching data.
2020-11-17 05:34:50 +00:00
Commit changes.
2021-02-24 09:52:29 +00:00
CompletableFuture<StreamInfo> futureStream = CompletableFuture.supplyAsync(() -> {
try {
return StreamInfo.getInfo("https://www.youtube.com/watch?v=" + videoId);
} catch (Exception e) {
ExceptionUtils.rethrow(e);
}
return null;
Fix LBRY timeouts.
2021-11-09 08:06:13 +00:00
}, Multithreading.getCachedExecutor());
Commit changes.
2021-02-24 09:52:29 +00:00
Add lbryId to streams response.
2021-08-08 11:17:58 +00:00
CompletableFuture<String> futureLbryId = CompletableFuture.supplyAsync(() -> {
try {
return getLBRYId(videoId);
} catch (Exception e) {
Don't fail hard when LBRY throws errors.
2021-10-06 15:37:30 +00:00
ExceptionHandler.handle(e);
Add lbryId to streams response.
2021-08-08 11:17:58 +00:00
}
return null;
Fix LBRY timeouts.
2021-11-09 08:06:13 +00:00
}, Multithreading.getCachedExecutor());
Add lbryId to streams response.
2021-08-08 11:17:58 +00:00
Commit changes.
2021-02-24 09:52:29 +00:00
CompletableFuture<String> futureLBRY = CompletableFuture.supplyAsync(() -> {
try {
Fix LBRY timeouts.
2021-11-09 08:06:13 +00:00
String lbryId = null;
lbryId = futureLbryId.completeOnTimeout(null, 2, TimeUnit.SECONDS).get();
return getLBRYStreamURL(lbryId);
Commit changes.
2021-02-24 09:52:29 +00:00
} catch (Exception e) {
Don't fail hard when LBRY throws errors.
2021-10-06 15:37:30 +00:00
ExceptionHandler.handle(e);
Commit changes.
2021-02-24 09:52:29 +00:00
}
return null;
Fix LBRY timeouts.
2021-11-09 08:06:13 +00:00
}, Multithreading.getCachedExecutor());
Commit changes.
2021-02-24 09:52:29 +00:00
final List<Subtitle> subtitles = new ObjectArrayList<>();
Implement chapter segment support. Closes https://github.com/TeamPiped/Piped/issues/600
2021-11-04 19:21:26 +00:00
final List<ChapterSegment> chapters = new ObjectArrayList<>();
Commit changes.
2021-02-24 09:52:29 +00:00
Allow early expiry of cookies. Currently configured to 30 minutes.
2021-05-28 15:17:25 +00:00
final StreamInfo info = futureStream.get();
Commit changes.
2021-02-24 09:52:29 +00:00
Implement chapter segment support. Closes https://github.com/TeamPiped/Piped/issues/600
2021-11-04 19:21:26 +00:00
info.getStreamSegments().forEach(segment -> {
chapters.add(
new ChapterSegment(segment.getTitle(), segment.getPreviewUrl(), segment.getStartTimeSeconds()));
});
Add more info to captions.
2021-04-21 14:22:58 +00:00
info.getSubtitles()
.forEach(subtitle -> subtitles.add(new Subtitle(rewriteURL(subtitle.getUrl()),
subtitle.getFormat().getMimeType(), subtitle.getDisplayLanguageName(),
subtitle.getLanguageTag(), subtitle.isAutoGenerated())));
Commit changes.
2021-02-24 09:52:29 +00:00
final List<PipedStream> videoStreams = new ObjectArrayList<>();
final List<PipedStream> audioStreams = new ObjectArrayList<>();
Add timeout for LBRY calls.
2021-10-12 10:44:21 +00:00
String lbryURL = null;
try {
Fix LBRY timeouts.
2021-11-09 08:06:13 +00:00
lbryURL = futureLBRY.completeOnTimeout(null, 3, TimeUnit.SECONDS).get();
Add timeout for LBRY calls.
2021-10-12 10:44:21 +00:00
} catch (Exception e) {
// ignored
}
Commit changes.
2021-02-24 09:52:29 +00:00
if (lbryURL != null)
videoStreams.add(new PipedStream(lbryURL, "MP4", "LBRY", "video/mp4", false));
Improve livestream detection.
2021-10-01 17:53:54 +00:00
boolean livestream = info.getStreamType() == StreamType.LIVE_STREAM;
Commit changes.
2021-02-24 09:52:29 +00:00
if (!livestream) {
info.getVideoOnlyStreams().forEach(stream -> videoStreams.add(new PipedStream(rewriteURL(stream.getUrl()),
String.valueOf(stream.getFormat()), stream.getResolution(), stream.getFormat().getMimeType(), true,
stream.getBitrate(), stream.getInitStart(), stream.getInitEnd(), stream.getIndexStart(),
stream.getIndexEnd(), stream.getCodec(), stream.getWidth(), stream.getHeight(), 30)));
info.getVideoStreams()
.forEach(stream -> videoStreams
.add(new PipedStream(rewriteURL(stream.getUrl()), String.valueOf(stream.getFormat()),
stream.getResolution(), stream.getFormat().getMimeType(), false)));
info.getAudioStreams()
.forEach(stream -> audioStreams.add(new PipedStream(rewriteURL(stream.getUrl()),
String.valueOf(stream.getFormat()), stream.getAverageBitrate() + " kbps",
stream.getFormat().getMimeType(), false, stream.getBitrate(), stream.getInitStart(),
stream.getInitEnd(), stream.getIndexStart(), stream.getIndexEnd(), stream.getCodec())));
}
final List<StreamItem> relatedStreams = new ObjectArrayList<>();
Refractor repeated code.
2021-09-09 19:25:16 +00:00
info.getRelatedItems().forEach(o -> relatedStreams.add(collectRelatedStream(o)));
Commit changes.
2021-02-24 09:52:29 +00:00
List<ChapterSegment> segments = new ObjectArrayList<>();
Implement chapter segment support. Closes https://github.com/TeamPiped/Piped/issues/600
2021-11-04 19:21:26 +00:00
info.getStreamSegments().forEach(segment -> segments
.add(new ChapterSegment(segment.getTitle(), segment.getPreviewUrl(), segment.getStartTimeSeconds())));
Commit changes.
2021-02-24 09:52:29 +00:00
Fix NPE.
2021-07-18 11:14:31 +00:00
long time = info.getUploadDate() != null ? info.getUploadDate().offsetDateTime().toInstant().toEpochMilli()
: System.currentTimeMillis();
Use Hibernate as a database (#32) * Basic hibernate structure. * Commit stash. * Improve a lot of things. * Implement Feed, PubSub, query optimizations. * Update video views on channel visits. * Allow mass importing subscriptions. * Allow configuring database in config.properties. * Major multi-threading and pubsub improvements. * PubSub query improvements and fix for postgres. * Revert docker-compose change. * Disable showing sql statements.
2021-07-16 22:40:46 +00:00
if (info.getUploadDate() != null && System.currentTimeMillis() - time < TimeUnit.DAYS.toMillis(10))
updateViews(info.getId(), info.getViewCount(), time, false);
Commit changes.
2021-02-24 09:52:29 +00:00
final Streams streams = new Streams(info.getName(), info.getDescription().getContent(),
Use substringAfter to cleanup code and fix YouTube music URLs.
2021-09-05 18:44:24 +00:00
info.getTextualUploadDate(), info.getUploaderName(), substringYouTube(info.getUploaderUrl()),
Commit changes.
2021-02-24 09:52:29 +00:00
rewriteURL(info.getUploaderAvatarUrl()), rewriteURL(info.getThumbnailUrl()), info.getDuration(),
Update NewPipeExtractor and extract more info.
2021-09-03 16:31:26 +00:00
info.getViewCount(), info.getLikeCount(), info.getDislikeCount(), info.isUploaderVerified(),
Improve livestream detection.
2021-10-01 17:53:54 +00:00
audioStreams, videoStreams, relatedStreams, subtitles, livestream, rewriteURL(info.getHlsUrl()),
Implement chapter segment support. Closes https://github.com/TeamPiped/Piped/issues/600
2021-11-04 19:21:26 +00:00
rewriteURL(info.getDashMpdUrl()), futureLbryId.get(), chapters);
Commit changes.
2021-02-24 09:52:29 +00:00
return Constants.mapper.writeValueAsBytes(streams);
Initial commit.
2020-11-12 21:19:45 +00:00
}
Support for initial pages of other channel URL types.
2021-05-28 18:40:11 +00:00
public static final byte[] channelResponse(String channelPath) throws Exception {
Initial commit.
2020-11-12 21:19:45 +00:00
Support for initial pages of other channel URL types.
2021-05-28 18:40:11 +00:00
final ChannelInfo info = ChannelInfo.getInfo("https://youtube.com/" + channelPath);
Initial commit.
2020-11-12 21:19:45 +00:00
Commit changes.
2021-02-24 09:52:29 +00:00
final List<StreamItem> relatedStreams = new ObjectArrayList<>();
Initial commit.
2020-11-12 21:19:45 +00:00
Refractor repeated code.
2021-09-09 19:25:16 +00:00
info.getRelatedItems().forEach(o -> relatedStreams.add(collectRelatedStream(o)));
Initial commit.
2020-11-12 21:19:45 +00:00
Use Hibernate as a database (#32) * Basic hibernate structure. * Commit stash. * Improve a lot of things. * Implement Feed, PubSub, query optimizations. * Update video views on channel visits. * Allow mass importing subscriptions. * Allow configuring database in config.properties. * Major multi-threading and pubsub improvements. * PubSub query improvements and fix for postgres. * Revert docker-compose change. * Disable showing sql statements.
2021-07-16 22:40:46 +00:00
Multithreading.runAsync(() -> {
Session s = DatabaseSessionFactory.createSession();
me.kavin.piped.utils.obj.db.Channel channel = DatabaseHelper.getChannelFromId(s, info.getId());
if (channel != null) {
Allow Avatar URL to change in channel.
2021-10-31 17:48:34 +00:00
if (channel.isVerified() != info.isVerified()
|| !channel.getUploaderAvatar().equals(info.getAvatarUrl())) {
Implement uploaderVerified for feed response.
2021-08-17 17:29:15 +00:00
channel.setVerified(info.isVerified());
Allow Avatar URL to change in channel.
2021-10-31 17:48:34 +00:00
channel.setUploaderAvatar(info.getAvatarUrl());
Implement uploaderVerified for feed response.
2021-08-17 17:29:15 +00:00
if (!s.getTransaction().isActive())
s.getTransaction().begin();
s.update(channel);
s.getTransaction().commit();
}
Use Hibernate as a database (#32) * Basic hibernate structure. * Commit stash. * Improve a lot of things. * Implement Feed, PubSub, query optimizations. * Update video views on channel visits. * Allow mass importing subscriptions. * Allow configuring database in config.properties. * Major multi-threading and pubsub improvements. * PubSub query improvements and fix for postgres. * Revert docker-compose change. * Disable showing sql statements.
2021-07-16 22:40:46 +00:00
for (StreamInfoItem item : info.getRelatedItems()) {
long time = item.getUploadDate() != null
? item.getUploadDate().offsetDateTime().toInstant().toEpochMilli()
: System.currentTimeMillis();
if (System.currentTimeMillis() - time < TimeUnit.DAYS.toMillis(10))
updateViews(item.getUrl().substring("https://www.youtube.com/watch?v=".length()),
item.getViewCount(), time, true);
}
}
s.close();
});
Cleanup page handling + fix channel continuations.
2021-06-05 19:34:33 +00:00
String nextpage = null;
ActiveJ migration. (#30) * Initial ActiveJ migration. * Add multi-threading.
2021-03-29 13:59:10 +00:00
if (info.hasNextPage()) {
Page page = info.getNextPage();
Cleanup page handling + fix channel continuations.
2021-06-05 19:34:33 +00:00
nextpage = Constants.mapper.writeValueAsString(page);
ActiveJ migration. (#30) * Initial ActiveJ migration. * Add multi-threading.
2021-03-29 13:59:10 +00:00
}
Channel Pages and search suggestions.
2020-11-25 05:26:25 +00:00
Commit changes.
2021-02-24 09:52:29 +00:00
final Channel channel = new Channel(info.getId(), info.getName(), rewriteURL(info.getAvatarUrl()),
Add verified info in more places.
2021-08-17 20:07:49 +00:00
rewriteURL(info.getBannerUrl()), info.getDescription(), info.getSubscriberCount(), info.isVerified(),
nextpage, relatedStreams);
Initial commit.
2020-11-12 21:19:45 +00:00
Commit changes.
2021-02-24 09:52:29 +00:00
IPFS.publishData(channel);
return Constants.mapper.writeValueAsBytes(channel);
Initial commit.
2020-11-12 21:19:45 +00:00
}
Cleanup page handling + fix channel continuations.
2021-06-05 19:34:33 +00:00
public static final byte[] channelPageResponse(String channelId, String prevpageStr)
Commit changes.
2021-02-24 09:52:29 +00:00
throws IOException, ExtractionException, InterruptedException {
Channel Pages and search suggestions.
2020-11-25 05:26:25 +00:00
Cleanup page handling + fix channel continuations.
2021-06-05 19:34:33 +00:00
Page prevpage = Constants.mapper.readValue(prevpageStr, Page.class);
Improve livestream detection.
2021-10-01 17:53:54 +00:00
InfoItemsPage<StreamInfoItem> info = ChannelInfo.getMoreItems(YOUTUBE_SERVICE,
Cleanup page handling + fix channel continuations.
2021-06-05 19:34:33 +00:00
"https://youtube.com/channel/" + channelId, prevpage);
Channel Pages and search suggestions.
2020-11-25 05:26:25 +00:00
Commit changes.
2021-02-24 09:52:29 +00:00
final List<StreamItem> relatedStreams = new ObjectArrayList<>();
Channel Pages and search suggestions.
2020-11-25 05:26:25 +00:00
Refractor repeated code.
2021-09-09 19:25:16 +00:00
info.getItems().forEach(o -> relatedStreams.add(collectRelatedStream(o)));
Channel Pages and search suggestions.
2020-11-25 05:26:25 +00:00
Cleanup page handling + fix channel continuations.
2021-06-05 19:34:33 +00:00
String nextpage = null;
Fix channel continuation.
2021-03-30 09:22:36 +00:00
if (info.hasNextPage()) {
Page page = info.getNextPage();
Cleanup page handling + fix channel continuations.
2021-06-05 19:34:33 +00:00
nextpage = Constants.mapper.writeValueAsString(page);
Fix channel continuation.
2021-03-30 09:22:36 +00:00
}
Channel Pages and search suggestions.
2020-11-25 05:26:25 +00:00
Cleanup page handling + fix channel continuations.
2021-06-05 19:34:33 +00:00
final StreamsPage streamspage = new StreamsPage(nextpage, relatedStreams);
Channel Pages and search suggestions.
2020-11-25 05:26:25 +00:00
Commit changes.
2021-02-24 09:52:29 +00:00
return Constants.mapper.writeValueAsBytes(streamspage);
Channel Pages and search suggestions.
2020-11-25 05:26:25 +00:00
}
Support region localization for trending (#79) * Support region localization for trending * Revert linting, remove defaulting to US, and fix warning. Co-authored-by: FireMasterK <20838718+FireMasterK@users.noreply.github.com>
2021-07-04 17:53:33 +00:00
public static final byte[] trendingResponse(String region)
throws ParsingException, ExtractionException, IOException {
Initial commit.
2020-11-12 21:19:45 +00:00
Handle invalid requests gracefully.
2021-07-20 21:03:44 +00:00
if (region == null)
return Constants.mapper.writeValueAsBytes(new InvalidRequestResponse());
Commit changes.
2021-02-24 09:52:29 +00:00
final List<StreamItem> relatedStreams = new ObjectArrayList<>();
Initial commit.
2020-11-12 21:19:45 +00:00
Improve livestream detection.
2021-10-01 17:53:54 +00:00
KioskList kioskList = YOUTUBE_SERVICE.getKioskList();
Support region localization for trending (#79) * Support region localization for trending * Revert linting, remove defaulting to US, and fix warning. Co-authored-by: FireMasterK <20838718+FireMasterK@users.noreply.github.com>
2021-07-04 17:53:33 +00:00
kioskList.forceContentCountry(new ContentCountry(region));
KioskExtractor<?> extractor = kioskList.getDefaultKioskExtractor();
extractor.fetchPage();
KioskInfo info = KioskInfo.getInfo(extractor);
Initial commit.
2020-11-12 21:19:45 +00:00
Refractor repeated code.
2021-09-09 19:25:16 +00:00
info.getRelatedItems().forEach(o -> relatedStreams.add(collectRelatedStream(o)));
Initial commit.
2020-11-12 21:19:45 +00:00
Commit changes.
2021-02-24 09:52:29 +00:00
return Constants.mapper.writeValueAsBytes(relatedStreams);
Initial commit.
2020-11-12 21:19:45 +00:00
}
performance: read and write data as byte arrays.
2021-01-12 08:15:09 +00:00
public static final byte[] playlistResponse(String playlistId)
Commit changes.
2021-02-24 09:52:29 +00:00
throws IOException, ExtractionException, InterruptedException {
add playlists and small changes.
2021-01-04 05:47:28 +00:00
Commit changes.
2021-02-24 09:52:29 +00:00
final PlaylistInfo info = PlaylistInfo.getInfo("https://www.youtube.com/playlist?list=" + playlistId);
add playlists and small changes.
2021-01-04 05:47:28 +00:00
Commit changes.
2021-02-24 09:52:29 +00:00
final List<StreamItem> relatedStreams = new ObjectArrayList<>();
add playlists and small changes.
2021-01-04 05:47:28 +00:00
Refractor repeated code.
2021-09-09 19:25:16 +00:00
info.getRelatedItems().forEach(o -> relatedStreams.add(collectRelatedStream(o)));
add playlists and small changes.
2021-01-04 05:47:28 +00:00
Cleanup page handling + fix channel continuations.
2021-06-05 19:34:33 +00:00
String nextpage = null;
Fix playlist continuation.
2021-03-30 09:36:01 +00:00
if (info.hasNextPage()) {
Page page = info.getNextPage();
Cleanup page handling + fix channel continuations.
2021-06-05 19:34:33 +00:00
nextpage = Constants.mapper.writeValueAsString(page);
Fix playlist continuation.
2021-03-30 09:36:01 +00:00
}
add playlists and small changes.
2021-01-04 05:47:28 +00:00
Commit changes.
2021-02-24 09:52:29 +00:00
final Playlist playlist = new Playlist(info.getName(), rewriteURL(info.getThumbnailUrl()),
Cleanup page handling + fix channel continuations.
2021-06-05 19:34:33 +00:00
rewriteURL(info.getBannerUrl()), nextpage,
Fix error when playlist has no uploader.
2021-06-01 18:52:48 +00:00
info.getUploaderName().isEmpty() ? null : info.getUploaderName(),
Use substringAfter to cleanup code and fix YouTube music URLs.
2021-09-05 18:44:24 +00:00
substringYouTube(info.getUploaderUrl()), rewriteURL(info.getUploaderAvatarUrl()),
(int) info.getStreamCount(), relatedStreams);
add playlists and small changes.
2021-01-04 05:47:28 +00:00
Commit changes.
2021-02-24 09:52:29 +00:00
return Constants.mapper.writeValueAsBytes(playlist);
add playlists and small changes.
2021-01-04 05:47:28 +00:00
}
Cleanup page handling + fix channel continuations.
2021-06-05 19:34:33 +00:00
public static final byte[] playlistPageResponse(String playlistId, String prevpageStr)
Commit changes.
2021-02-24 09:52:29 +00:00
throws IOException, ExtractionException, InterruptedException {
add playlists and small changes.
2021-01-04 05:47:28 +00:00
Cleanup page handling + fix channel continuations.
2021-06-05 19:34:33 +00:00
Page prevpage = Constants.mapper.readValue(prevpageStr, Page.class);
Improve livestream detection.
2021-10-01 17:53:54 +00:00
InfoItemsPage<StreamInfoItem> info = PlaylistInfo.getMoreItems(YOUTUBE_SERVICE,
Cleanup page handling + fix channel continuations.
2021-06-05 19:34:33 +00:00
"https://www.youtube.com/playlist?list=" + playlistId, prevpage);
add playlists and small changes.
2021-01-04 05:47:28 +00:00
Commit changes.
2021-02-24 09:52:29 +00:00
final List<StreamItem> relatedStreams = new ObjectArrayList<>();
add playlists and small changes.
2021-01-04 05:47:28 +00:00
Refractor repeated code.
2021-09-09 19:25:16 +00:00
info.getItems().forEach(o -> relatedStreams.add(collectRelatedStream(o)));
add playlists and small changes.
2021-01-04 05:47:28 +00:00
Cleanup page handling + fix channel continuations.
2021-06-05 19:34:33 +00:00
String nextpage = null;
Fix playlist continuation.
2021-03-30 09:36:01 +00:00
if (info.hasNextPage()) {
Page page = info.getNextPage();
Cleanup page handling + fix channel continuations.
2021-06-05 19:34:33 +00:00
nextpage = Constants.mapper.writeValueAsString(page);
Fix playlist continuation.
2021-03-30 09:36:01 +00:00
}
add playlists and small changes.
2021-01-04 05:47:28 +00:00
Cleanup page handling + fix channel continuations.
2021-06-05 19:34:33 +00:00
final StreamsPage streamspage = new StreamsPage(nextpage, relatedStreams);
add playlists and small changes.
2021-01-04 05:47:28 +00:00
Commit changes.
2021-02-24 09:52:29 +00:00
return Constants.mapper.writeValueAsBytes(streamspage);
add playlists and small changes.
2021-01-04 05:47:28 +00:00
}
Add Playlist RSS (#80) * WIP Playlist rss. * Complete adding support.
2021-07-04 18:57:18 +00:00
public static final byte[] playlistRSSResponse(String playlistId)
throws IOException, ExtractionException, InterruptedException, FeedException {
final PlaylistInfo info = PlaylistInfo.getInfo("https://www.youtube.com/playlist?list=" + playlistId);
final List<SyndEntry> entries = new ObjectArrayList<>();
SyndFeed feed = new SyndFeedImpl();
feed.setFeedType("rss_2.0");
feed.setTitle(info.getName());
feed.setAuthor(info.getUploaderName());
feed.setLink(info.getUrl());
feed.setDescription(String.format("%s - Piped", info.getName()));
info.getRelatedItems().forEach(o -> {
StreamInfoItem item = o;
SyndEntry entry = new SyndEntryImpl();
entry.setAuthor(item.getUploaderName());
Change rss entry from id to link (#150) Uri -> Link
2021-12-20 14:36:20 +00:00
entry.setLink(item.getUrl());
Add Playlist RSS (#80) * WIP Playlist rss. * Complete adding support.
2021-07-04 18:57:18 +00:00
entry.setTitle(item.getName());
entries.add(entry);
});
feed.setEntries(entries);
return new SyndFeedOutput().outputString(feed).getBytes(StandardCharsets.UTF_8);
}
performance: read and write data as byte arrays.
2021-01-12 08:15:09 +00:00
public static final byte[] suggestionsResponse(String query)
Commit changes.
2021-02-24 09:52:29 +00:00
throws JsonProcessingException, IOException, ExtractionException {
Channel Pages and search suggestions.
2020-11-25 05:26:25 +00:00
Improve livestream detection.
2021-10-01 17:53:54 +00:00
return Constants.mapper.writeValueAsBytes(YOUTUBE_SERVICE.getSuggestionExtractor().suggestionList(query));
Channel Pages and search suggestions.
2020-11-25 05:26:25 +00:00
}
Implement OpenSearch api.
2021-10-25 22:55:29 +00:00
public static final byte[] opensearchSuggestionsResponse(String query)
throws JsonProcessingException, IOException, ExtractionException {
return Constants.mapper.writeValueAsBytes(
Arrays.asList(query, YOUTUBE_SERVICE.getSuggestionExtractor().suggestionList(query)));
}
Add support for search filters and simplify continuations.
2021-06-14 19:44:23 +00:00
public static final byte[] searchResponse(String q, String filter)
throws IOException, ExtractionException, InterruptedException {
Add search.
2020-12-09 13:30:42 +00:00
Improve livestream detection.
2021-10-01 17:53:54 +00:00
final SearchInfo info = SearchInfo.getInfo(YOUTUBE_SERVICE,
YOUTUBE_SERVICE.getSearchQHFactory().fromQuery(q, Collections.singletonList(filter), null));
Commit changes.
2021-02-24 09:52:29 +00:00
Unify video/StreamInfoItem in Search Results.
2021-09-03 19:32:51 +00:00
ObjectArrayList<Object> items = new ObjectArrayList<>();
Commit changes.
2021-02-24 09:52:29 +00:00
info.getRelatedItems().forEach(item -> {
switch (item.getInfoType()) {
case STREAM:
Refractor repeated code.
2021-09-09 19:25:16 +00:00
items.add(collectRelatedStream(item));
Commit changes.
2021-02-24 09:52:29 +00:00
break;
case CHANNEL:
Add additional details and playlists on search endpoint.
2021-05-28 12:08:32 +00:00
ChannelInfoItem channel = (ChannelInfoItem) item;
items.add(new SearchChannel(item.getName(), rewriteURL(item.getThumbnailUrl()),
Use substringAfter to cleanup code and fix YouTube music URLs.
2021-09-05 18:44:24 +00:00
substringYouTube(item.getUrl()), channel.getDescription(), channel.getSubscriberCount(),
Add additional details and playlists on search endpoint.
2021-05-28 12:08:32 +00:00
channel.getStreamCount(), channel.isVerified()));
break;
case PLAYLIST:
PlaylistInfoItem playlist = (PlaylistInfoItem) item;
items.add(new SearchPlaylist(item.getName(), rewriteURL(item.getThumbnailUrl()),
Use substringAfter to cleanup code and fix YouTube music URLs.
2021-09-05 18:44:24 +00:00
substringYouTube(item.getUrl()), playlist.getUploaderName(), playlist.getStreamCount()));
Commit changes.
2021-02-24 09:52:29 +00:00
break;
default:
break;
}
});
Page nextpage = info.getNextPage();
Include search corrections in API.
2021-10-09 23:24:29 +00:00
return Constants.mapper.writeValueAsBytes(new SearchResults(items,
Constants.mapper.writeValueAsString(nextpage), info.getSearchSuggestion(), info.isCorrectedSearch()));
Next page in search results.
2020-12-14 07:11:42 +00:00
}
Add support for search filters and simplify continuations.
2021-06-14 19:44:23 +00:00
public static final byte[] searchPageResponse(String q, String filter, String prevpageStr)
Commit changes.
2021-02-24 09:52:29 +00:00
throws IOException, ExtractionException, InterruptedException {
Add support for search filters and simplify continuations.
2021-06-14 19:44:23 +00:00
Page prevpage = Constants.mapper.readValue(prevpageStr, Page.class);
Improve livestream detection.
2021-10-01 17:53:54 +00:00
InfoItemsPage<InfoItem> pages = SearchInfo.getMoreItems(YOUTUBE_SERVICE,
YOUTUBE_SERVICE.getSearchQHFactory().fromQuery(q, Collections.singletonList(filter), null), prevpage);
Commit changes.
2021-02-24 09:52:29 +00:00
Unify video/StreamInfoItem in Search Results.
2021-09-03 19:32:51 +00:00
ObjectArrayList<Object> items = new ObjectArrayList<>();
Commit changes.
2021-02-24 09:52:29 +00:00
pages.getItems().forEach(item -> {
switch (item.getInfoType()) {
case STREAM:
Refractor repeated code.
2021-09-09 19:25:16 +00:00
items.add(collectRelatedStream(item));
Commit changes.
2021-02-24 09:52:29 +00:00
break;
case CHANNEL:
Add additional details and playlists on search endpoint.
2021-05-28 12:08:32 +00:00
ChannelInfoItem channel = (ChannelInfoItem) item;
items.add(new SearchChannel(item.getName(), rewriteURL(item.getThumbnailUrl()),
Use substringAfter to cleanup code and fix YouTube music URLs.
2021-09-05 18:44:24 +00:00
substringYouTube(item.getUrl()), channel.getDescription(), channel.getSubscriberCount(),
Add additional details and playlists on search endpoint.
2021-05-28 12:08:32 +00:00
channel.getStreamCount(), channel.isVerified()));
break;
case PLAYLIST:
PlaylistInfoItem playlist = (PlaylistInfoItem) item;
items.add(new SearchPlaylist(item.getName(), rewriteURL(item.getThumbnailUrl()),
Use substringAfter to cleanup code and fix YouTube music URLs.
2021-09-05 18:44:24 +00:00
substringYouTube(item.getUrl()), playlist.getUploaderName(), playlist.getStreamCount()));
Commit changes.
2021-02-24 09:52:29 +00:00
break;
default:
break;
}
});
Page nextpage = pages.getNextPage();
Add support for search filters and simplify continuations.
2021-06-14 19:44:23 +00:00
return Constants.mapper
.writeValueAsBytes(new SearchResults(items, Constants.mapper.writeValueAsString(nextpage)));
Add search.
2020-12-09 13:30:42 +00:00
}
Add comments API.
2021-04-03 15:08:32 +00:00
public static final byte[] commentsResponse(String videoId) throws Exception {
CommentsInfo info = commentsCache.get(videoId);
List<Comment> comments = new ObjectArrayList<>();
info.getRelatedItems().forEach(comment -> {
Add support for extracting replies and update NPE.
2021-09-22 14:18:24 +00:00
try {
Fix for Comment repliesPage being "null" when not present.
2021-11-01 19:58:44 +00:00
String repliespage = null;
if (comment.getReplies() != null)
repliespage = Constants.mapper.writeValueAsString(comment.getReplies());
Add support for extracting replies and update NPE.
2021-09-22 14:18:24 +00:00
comments.add(new Comment(comment.getUploaderName(), rewriteURL(comment.getUploaderAvatarUrl()),
comment.getCommentId(), comment.getCommentText(), comment.getTextualUploadDate(),
Fix for Comment repliesPage being "null" when not present.
2021-11-01 19:58:44 +00:00
substringYouTube(comment.getUploaderUrl()), repliespage, comment.getLikeCount(),
Add support for extracting replies and update NPE.
2021-09-22 14:18:24 +00:00
comment.isHeartedByUploader(), comment.isPinned(), comment.isUploaderVerified()));
} catch (JsonProcessingException e) {
ExceptionHandler.handle(e);
}
Add comments API.
2021-04-03 15:08:32 +00:00
});
String nextpage = null;
Update NPE fork and fix comments.
2021-07-29 19:21:37 +00:00
if (info.hasNextPage()) {
Page page = info.getNextPage();
nextpage = Constants.mapper.writeValueAsString(page);
}
Add comments API.
2021-04-03 15:08:32 +00:00
Fix NPE and add comments disabled field.
2021-07-31 10:39:06 +00:00
CommentsPage commentsItem = new CommentsPage(comments, nextpage, info.isCommentsDisabled());
Add comments API.
2021-04-03 15:08:32 +00:00
return Constants.mapper.writeValueAsBytes(commentsItem);
}
Update NPE fork and fix comments.
2021-07-29 19:21:37 +00:00
public static final byte[] commentsPageResponse(String videoId, String prevpageStr) throws Exception {
Page prevpage = Constants.mapper.readValue(prevpageStr, Page.class);
Add comments API.
2021-04-03 15:08:32 +00:00
CommentsInfo init = commentsCache.get(videoId);
Update NPE fork and fix comments.
2021-07-29 19:21:37 +00:00
InfoItemsPage<CommentsInfoItem> info = CommentsInfo.getMoreItems(init, prevpage);
Add comments API.
2021-04-03 15:08:32 +00:00
List<Comment> comments = new ObjectArrayList<>();
info.getItems().forEach(comment -> {
Add support for extracting replies and update NPE.
2021-09-22 14:18:24 +00:00
try {
Fix for Comment repliesPage being "null" when not present.
2021-11-01 19:58:44 +00:00
String repliespage = null;
if (comment.getReplies() != null)
repliespage = Constants.mapper.writeValueAsString(comment.getReplies());
Add support for extracting replies and update NPE.
2021-09-22 14:18:24 +00:00
comments.add(new Comment(comment.getUploaderName(), rewriteURL(comment.getUploaderAvatarUrl()),
comment.getCommentId(), comment.getCommentText(), comment.getTextualUploadDate(),
Fix for Comment repliesPage being "null" when not present.
2021-11-01 19:58:44 +00:00
substringYouTube(comment.getUploaderUrl()), repliespage, comment.getLikeCount(),
Add support for extracting replies and update NPE.
2021-09-22 14:18:24 +00:00
comment.isHeartedByUploader(), comment.isPinned(), comment.isUploaderVerified()));
} catch (JsonProcessingException e) {
ExceptionHandler.handle(e);
}
Add comments API.
2021-04-03 15:08:32 +00:00
});
String nextpage = null;
Update NPE fork and fix comments.
2021-07-29 19:21:37 +00:00
if (info.hasNextPage()) {
Page page = info.getNextPage();
nextpage = Constants.mapper.writeValueAsString(page);
}
Add comments API.
2021-04-03 15:08:32 +00:00
Fix NPE and add comments disabled field.
2021-07-31 10:39:06 +00:00
CommentsPage commentsItem = new CommentsPage(comments, nextpage, init.isCommentsDisabled());
Add comments API.
2021-04-03 15:08:32 +00:00
return Constants.mapper.writeValueAsBytes(commentsItem);
}
Use Hibernate as a database (#32) * Basic hibernate structure. * Commit stash. * Improve a lot of things. * Implement Feed, PubSub, query optimizations. * Update video views on channel visits. * Allow mass importing subscriptions. * Allow configuring database in config.properties. * Major multi-threading and pubsub improvements. * PubSub query improvements and fix for postgres. * Revert docker-compose change. * Disable showing sql statements.
2021-07-16 22:40:46 +00:00
private static final Argon2PasswordEncoder argon2PasswordEncoder = new Argon2PasswordEncoder();
Check if passwords are compromised in HIBP.
2021-09-05 19:19:54 +00:00
public static final byte[] registerResponse(String user, String pass) throws IOException, NoSuchAlgorithmException,
InvalidKeySpecException, InterruptedException, URISyntaxException {
Use Hibernate as a database (#32) * Basic hibernate structure. * Commit stash. * Improve a lot of things. * Implement Feed, PubSub, query optimizations. * Update video views on channel visits. * Allow mass importing subscriptions. * Allow configuring database in config.properties. * Major multi-threading and pubsub improvements. * PubSub query improvements and fix for postgres. * Revert docker-compose change. * Disable showing sql statements.
2021-07-16 22:40:46 +00:00
Add configuration property to disable registration.
2021-11-09 06:27:29 +00:00
if (Constants.DISABLE_REGISTRATION)
return Constants.mapper.writeValueAsBytes(new DisabledRegistrationResponse());
Handle invalid requests gracefully.
2021-07-20 21:03:44 +00:00
if (user == null || pass == null)
return Constants.mapper.writeValueAsBytes(new InvalidRequestResponse());
Use Hibernate as a database (#32) * Basic hibernate structure. * Commit stash. * Improve a lot of things. * Implement Feed, PubSub, query optimizations. * Update video views on channel visits. * Allow mass importing subscriptions. * Allow configuring database in config.properties. * Major multi-threading and pubsub improvements. * PubSub query improvements and fix for postgres. * Revert docker-compose change. * Disable showing sql statements.
2021-07-16 22:40:46 +00:00
user = user.toLowerCase();
Session s = DatabaseSessionFactory.createSession();
CriteriaBuilder cb = s.getCriteriaBuilder();
CriteriaQuery<User> cr = cb.createQuery(User.class);
Root<User> root = cr.from(User.class);
cr.select(root).where(root.get("username").in(user));
boolean registered = s.createQuery(cr).uniqueResult() != null;
if (registered) {
s.close();
return Constants.mapper.writeValueAsBytes(new AlreadyRegisteredResponse());
}
Add option to disable HIBP password check. Usage of this option is discouraged for password security reason.
2021-10-04 18:57:56 +00:00
if (Constants.COMPROMISED_PASSWORD_CHECK) {
Check if passwords are compromised in HIBP.
2021-09-05 19:19:54 +00:00
String sha1Hash = DigestUtils.sha1Hex(pass).toUpperCase();
String prefix = sha1Hash.substring(0, 5);
String suffix = sha1Hash.substring(5);
String[] entries = RequestUtils
.sendGet("https://api.pwnedpasswords.com/range/" + prefix, "github.com/TeamPiped/Piped-Backend")
.split("\n");
for (String entry : entries)
if (StringUtils.substringBefore(entry, ":").equals(suffix))
return Constants.mapper.writeValueAsBytes(new CompromisedPasswordResponse());
}
Use Hibernate as a database (#32) * Basic hibernate structure. * Commit stash. * Improve a lot of things. * Implement Feed, PubSub, query optimizations. * Update video views on channel visits. * Allow mass importing subscriptions. * Allow configuring database in config.properties. * Major multi-threading and pubsub improvements. * PubSub query improvements and fix for postgres. * Revert docker-compose change. * Disable showing sql statements.
2021-07-16 22:40:46 +00:00
User newuser = new User(user, argon2PasswordEncoder.encode(pass), Collections.emptyList());
s.save(newuser);
s.getTransaction().begin();
s.getTransaction().commit();
s.close();
return Constants.mapper.writeValueAsBytes(new LoginResponse(newuser.getSessionId()));
}
Add support for legacy bcrypt hashes from Invidious
2021-10-03 08:02:01 +00:00
private static final BCryptPasswordEncoder bcryptPasswordEncoder = new BCryptPasswordEncoder();
Use Hibernate as a database (#32) * Basic hibernate structure. * Commit stash. * Improve a lot of things. * Implement Feed, PubSub, query optimizations. * Update video views on channel visits. * Allow mass importing subscriptions. * Allow configuring database in config.properties. * Major multi-threading and pubsub improvements. * PubSub query improvements and fix for postgres. * Revert docker-compose change. * Disable showing sql statements.
2021-07-16 22:40:46 +00:00
public static final byte[] loginResponse(String user, String pass)
throws IOException, NoSuchAlgorithmException, InvalidKeySpecException {
Handle invalid requests gracefully.
2021-07-20 21:03:44 +00:00
if (user == null || pass == null)
return Constants.mapper.writeValueAsBytes(new InvalidRequestResponse());
Use Hibernate as a database (#32) * Basic hibernate structure. * Commit stash. * Improve a lot of things. * Implement Feed, PubSub, query optimizations. * Update video views on channel visits. * Allow mass importing subscriptions. * Allow configuring database in config.properties. * Major multi-threading and pubsub improvements. * PubSub query improvements and fix for postgres. * Revert docker-compose change. * Disable showing sql statements.
2021-07-16 22:40:46 +00:00
user = user.toLowerCase();
Session s = DatabaseSessionFactory.createSession();
CriteriaBuilder cb = s.getCriteriaBuilder();
CriteriaQuery<User> cr = cb.createQuery(User.class);
Root<User> root = cr.from(User.class);
cr.select(root).where(root.get("username").in(user));
User dbuser = s.createQuery(cr).uniqueResult();
Add support for legacy bcrypt hashes from Invidious
2021-10-03 08:02:01 +00:00
if (dbuser != null) {
Fix for possible NPE.
2021-10-03 08:55:13 +00:00
String hash = dbuser.getPassword();
if (hash.startsWith("$argon2")) {
if (argon2PasswordEncoder.matches(pass, hash)) {
s.close();
return Constants.mapper.writeValueAsBytes(new LoginResponse(dbuser.getSessionId()));
}
} else if (bcryptPasswordEncoder.matches(pass, hash)) {
Add support for legacy bcrypt hashes from Invidious
2021-10-03 08:02:01 +00:00
s.close();
return Constants.mapper.writeValueAsBytes(new LoginResponse(dbuser.getSessionId()));
}
Use Hibernate as a database (#32) * Basic hibernate structure. * Commit stash. * Improve a lot of things. * Implement Feed, PubSub, query optimizations. * Update video views on channel visits. * Allow mass importing subscriptions. * Allow configuring database in config.properties. * Major multi-threading and pubsub improvements. * PubSub query improvements and fix for postgres. * Revert docker-compose change. * Disable showing sql statements.
2021-07-16 22:40:46 +00:00
}
s.close();
return Constants.mapper.writeValueAsBytes(new IncorrectCredentialsResponse());
}
public static final byte[] subscribeResponse(String session, String channelId)
throws IOException, NoSuchAlgorithmException, InvalidKeySpecException {
Session s = DatabaseSessionFactory.createSession();
User user = DatabaseHelper.getUserFromSessionWithSubscribed(s, session);
if (user != null) {
if (!user.getSubscribed().contains(channelId)) {
s.getTransaction().begin();
s.createNativeQuery("insert into users_subscribed (subscriber, channel) values (?,?)")
.setParameter(1, user.getId()).setParameter(2, channelId).executeUpdate();
s.getTransaction().commit();
s.close();
Multithreading.runAsync(() -> {
Session sess = DatabaseSessionFactory.createSession();
me.kavin.piped.utils.obj.db.Channel channel = DatabaseHelper.getChannelFromId(sess, channelId);
if (channel == null) {
ChannelInfo info = null;
try {
info = ChannelInfo.getInfo("https://youtube.com/channel/" + channelId);
} catch (IOException | ExtractionException e) {
ExceptionUtils.rethrow(e);
}
channel = new me.kavin.piped.utils.obj.db.Channel(channelId, info.getName(),
Implement uploaderVerified for feed response.
2021-08-17 17:29:15 +00:00
info.getAvatarUrl(), info.isVerified());
Use Hibernate as a database (#32) * Basic hibernate structure. * Commit stash. * Improve a lot of things. * Implement Feed, PubSub, query optimizations. * Update video views on channel visits. * Allow mass importing subscriptions. * Allow configuring database in config.properties. * Major multi-threading and pubsub improvements. * PubSub query improvements and fix for postgres. * Revert docker-compose change. * Disable showing sql statements.
2021-07-16 22:40:46 +00:00
sess.save(channel);
sess.beginTransaction().commit();
Print stacktraces on any errors.
2021-07-21 12:37:43 +00:00
Multithreading.runAsync(() -> {
try {
Session sessSub = DatabaseSessionFactory.createSession();
subscribePubSub(channelId, sessSub);
sessSub.close();
} catch (Exception e) {
Refractor and improve log handling.
2021-09-09 18:49:46 +00:00
ExceptionHandler.handle(e);
Print stacktraces on any errors.
2021-07-21 12:37:43 +00:00
}
});
Use Hibernate as a database (#32) * Basic hibernate structure. * Commit stash. * Improve a lot of things. * Implement Feed, PubSub, query optimizations. * Update video views on channel visits. * Allow mass importing subscriptions. * Allow configuring database in config.properties. * Major multi-threading and pubsub improvements. * PubSub query improvements and fix for postgres. * Revert docker-compose change. * Disable showing sql statements.
2021-07-16 22:40:46 +00:00
for (StreamInfoItem item : info.getRelatedItems()) {
long time = item.getUploadDate() != null
? item.getUploadDate().offsetDateTime().toInstant().toEpochMilli()
: System.currentTimeMillis();
if ((System.currentTimeMillis() - time) < TimeUnit.DAYS.toMillis(10))
Fix feed updates on manual subscription.
2021-07-17 14:02:57 +00:00
handleNewVideo(item.getUrl(), time, channel, sess);
Use Hibernate as a database (#32) * Basic hibernate structure. * Commit stash. * Improve a lot of things. * Implement Feed, PubSub, query optimizations. * Update video views on channel visits. * Allow mass importing subscriptions. * Allow configuring database in config.properties. * Major multi-threading and pubsub improvements. * PubSub query improvements and fix for postgres. * Revert docker-compose change. * Disable showing sql statements.
2021-07-16 22:40:46 +00:00
}
}
sess.close();
});
}
return Constants.mapper.writeValueAsBytes(new AcceptedResponse());
}
s.close();
return Constants.mapper.writeValueAsBytes(new AuthenticationFailureResponse());
}
public static final byte[] unsubscribeResponse(String session, String channelId)
throws IOException, NoSuchAlgorithmException, InvalidKeySpecException {
Session s = DatabaseSessionFactory.createSession();
User user = DatabaseHelper.getUserFromSession(s, session);
if (user != null) {
s.getTransaction().begin();
s.createNativeQuery("delete from users_subscribed where subscriber = :id and channel = :channel")
.setParameter("id", user.getId()).setParameter("channel", channelId).executeUpdate();
s.getTransaction().commit();
s.close();
return Constants.mapper.writeValueAsBytes(new AcceptedResponse());
}
s.close();
return Constants.mapper.writeValueAsBytes(new AuthenticationFailureResponse());
}
public static final byte[] isSubscribedResponse(String session, String channelId)
throws IOException, NoSuchAlgorithmException, InvalidKeySpecException {
Session s = DatabaseSessionFactory.createSession();
User user = DatabaseHelper.getUserFromSessionWithSubscribed(s, session);
if (user != null) {
if (user.getSubscribed().contains(channelId)) {
s.close();
return Constants.mapper.writeValueAsBytes(new SubscribeStatusResponse(true));
}
s.close();
return Constants.mapper.writeValueAsBytes(new SubscribeStatusResponse(false));
}
s.close();
return Constants.mapper.writeValueAsBytes(new AuthenticationFailureResponse());
}
public static final byte[] feedResponse(String session)
throws IOException, NoSuchAlgorithmException, InvalidKeySpecException {
Session s = DatabaseSessionFactory.createSession();
Improve feed query performance.
2021-09-21 21:35:12 +00:00
User user = DatabaseHelper.getUserFromSession(s, session);
Use Hibernate as a database (#32) * Basic hibernate structure. * Commit stash. * Improve a lot of things. * Implement Feed, PubSub, query optimizations. * Update video views on channel visits. * Allow mass importing subscriptions. * Allow configuring database in config.properties. * Major multi-threading and pubsub improvements. * PubSub query improvements and fix for postgres. * Revert docker-compose change. * Disable showing sql statements.
2021-07-16 22:40:46 +00:00
if (user != null) {
Improve feed query performance.
2021-09-21 21:35:12 +00:00
@SuppressWarnings("unchecked")
List<Object[]> queryResults = s.createNativeQuery(
"select Video.*, Channel.* from videos as Video left join channels as Channel on Video.uploader_id = Channel.uploader_id inner join users_subscribed on users_subscribed.channel = Channel.uploader_id where users_subscribed.subscriber = :user")
.setParameter("user", user.getId()).addEntity("Video", Video.class)
.addEntity("Channel", me.kavin.piped.utils.obj.db.Channel.class).getResultList();
Use Hibernate as a database (#32) * Basic hibernate structure. * Commit stash. * Improve a lot of things. * Implement Feed, PubSub, query optimizations. * Update video views on channel visits. * Allow mass importing subscriptions. * Allow configuring database in config.properties. * Major multi-threading and pubsub improvements. * PubSub query improvements and fix for postgres. * Revert docker-compose change. * Disable showing sql statements.
2021-07-16 22:40:46 +00:00
List<FeedItem> feedItems = new ObjectArrayList<>();
Improve feed query performance.
2021-09-21 21:35:12 +00:00
queryResults.forEach(obj -> {
Video video = (Video) obj[0];
me.kavin.piped.utils.obj.db.Channel channel = (me.kavin.piped.utils.obj.db.Channel) obj[1];
add playlists and small changes.
2021-01-04 05:47:28 +00:00
Improve feed query performance.
2021-09-21 21:35:12 +00:00
feedItems.add(new FeedItem("/watch?v=" + video.getId(), video.getTitle(),
rewriteURL(video.getThumbnail()), "/channel/" + channel.getUploaderId(), channel.getUploader(),
rewriteURL(channel.getUploaderAvatar()), video.getViews(), video.getDuration(),
video.getUploaded(), channel.isVerified()));
Use Hibernate as a database (#32) * Basic hibernate structure. * Commit stash. * Improve a lot of things. * Implement Feed, PubSub, query optimizations. * Update video views on channel visits. * Allow mass importing subscriptions. * Allow configuring database in config.properties. * Major multi-threading and pubsub improvements. * PubSub query improvements and fix for postgres. * Revert docker-compose change. * Disable showing sql statements.
2021-07-16 22:40:46 +00:00
Improve feed query performance.
2021-09-21 21:35:12 +00:00
});
Use Hibernate as a database (#32) * Basic hibernate structure. * Commit stash. * Improve a lot of things. * Implement Feed, PubSub, query optimizations. * Update video views on channel visits. * Allow mass importing subscriptions. * Allow configuring database in config.properties. * Major multi-threading and pubsub improvements. * PubSub query improvements and fix for postgres. * Revert docker-compose change. * Disable showing sql statements.
2021-07-16 22:40:46 +00:00
Improve feed query performance.
2021-09-21 21:35:12 +00:00
Collections.sort(feedItems, (a, b) -> (int) (b.uploaded - a.uploaded));
Use Hibernate as a database (#32) * Basic hibernate structure. * Commit stash. * Improve a lot of things. * Implement Feed, PubSub, query optimizations. * Update video views on channel visits. * Allow mass importing subscriptions. * Allow configuring database in config.properties. * Major multi-threading and pubsub improvements. * PubSub query improvements and fix for postgres. * Revert docker-compose change. * Disable showing sql statements.
2021-07-16 22:40:46 +00:00
s.close();
return Constants.mapper.writeValueAsBytes(feedItems);
}
s.close();
return Constants.mapper.writeValueAsBytes(new AuthenticationFailureResponse());
}
Generate RSS feed for subscriptions.
2021-07-19 19:44:18 +00:00
public static final byte[] feedResponseRSS(String session)
throws IOException, NoSuchAlgorithmException, InvalidKeySpecException, FeedException {
Session s = DatabaseSessionFactory.createSession();
Improve feed query performance.
2021-09-21 21:35:12 +00:00
User user = DatabaseHelper.getUserFromSession(s, session);
Generate RSS feed for subscriptions.
2021-07-19 19:44:18 +00:00
if (user != null) {
SyndFeed feed = new SyndFeedImpl();
feed.setFeedType("atom_1.0");
feed.setTitle("Piped - Feed");
feed.setDescription(String.format("Piped's RSS subscription feed for %s.", user.getUsername()));
Add FRONTEND_URL configuration property.
2021-09-09 08:05:19 +00:00
feed.setUri(Constants.FRONTEND_URL + "/feed");
Generate RSS feed for subscriptions.
2021-07-19 19:44:18 +00:00
if (user.getSubscribed() != null && !user.getSubscribed().isEmpty()) {
Improve feed query performance.
2021-09-21 21:35:12 +00:00
@SuppressWarnings("unchecked")
List<Object[]> queryResults = s.createNativeQuery(
"select Video.*, Channel.* from videos as Video left join channels as Channel on Video.uploader_id = Channel.uploader_id inner join users_subscribed on users_subscribed.channel = Channel.uploader_id where users_subscribed.subscriber = :user")
.setParameter("user", user.getId()).addEntity("Video", Video.class)
.addEntity("Channel", me.kavin.piped.utils.obj.db.Channel.class).getResultList();
Generate RSS feed for subscriptions.
2021-07-19 19:44:18 +00:00
Improve feed query performance.
2021-09-21 21:35:12 +00:00
Collections.sort(queryResults,
(a, b) -> (int) (((Video) b[0]).getUploaded() - ((Video) a[0]).getUploaded()));
Generate RSS feed for subscriptions.
2021-07-19 19:44:18 +00:00
final List<SyndEntry> entries = new ObjectArrayList<>();
Improve feed query performance.
2021-09-21 21:35:12 +00:00
for (Object[] result : queryResults) {
Video video = (Video) result[0];
me.kavin.piped.utils.obj.db.Channel channel = (me.kavin.piped.utils.obj.db.Channel) result[1];
Generate RSS feed for subscriptions.
2021-07-19 19:44:18 +00:00
SyndEntry entry = new SyndEntryImpl();
SyndPerson person = new SyndPersonImpl();
Improve feed query performance.
2021-09-21 21:35:12 +00:00
person.setName(channel.getUploader());
person.setUri(Constants.FRONTEND_URL + "/channel/" + channel.getUploaderId());
Generate RSS feed for subscriptions.
2021-07-19 19:44:18 +00:00
entry.setAuthors(Collections.singletonList(person));
Change rss entry from id to link (#150) Uri -> Link
2021-12-20 14:36:20 +00:00
entry.setLink(Constants.FRONTEND_URL + "/watch?v=" + video.getId());
Generate RSS feed for subscriptions.
2021-07-19 19:44:18 +00:00
entry.setTitle(video.getTitle());
entry.setPublishedDate(new Date(video.getUploaded()));
entries.add(entry);
}
feed.setEntries(entries);
}
s.close();
return new SyndFeedOutput().outputString(feed).getBytes(StandardCharsets.UTF_8);
}
s.close();
return Constants.mapper.writeValueAsBytes(new AuthenticationFailureResponse());
}
Add support for overriding subscriptions when importing.
2021-08-28 07:56:50 +00:00
public static final byte[] importResponse(String session, String[] channelIds, boolean override)
Use Hibernate as a database (#32) * Basic hibernate structure. * Commit stash. * Improve a lot of things. * Implement Feed, PubSub, query optimizations. * Update video views on channel visits. * Allow mass importing subscriptions. * Allow configuring database in config.properties. * Major multi-threading and pubsub improvements. * PubSub query improvements and fix for postgres. * Revert docker-compose change. * Disable showing sql statements.
2021-07-16 22:40:46 +00:00
throws IOException, NoSuchAlgorithmException, InvalidKeySpecException {
Session s = DatabaseSessionFactory.createSession();
User user = DatabaseHelper.getUserFromSessionWithSubscribed(s, session);
if (user != null) {
Multithreading.runAsync(() -> {
Revert "Fix LBRY timeouts." This reverts commit a7b02e9566e72f6ea67169fc618e9dbf0483e002.
2021-11-09 08:03:21 +00:00
if (override)
user.setSubscribed(Arrays.asList(channelIds));
else
Add support for overriding subscriptions when importing.
2021-08-28 07:56:50 +00:00
for (String channelId : channelIds)
Revert "Fix LBRY timeouts." This reverts commit a7b02e9566e72f6ea67169fc618e9dbf0483e002.
2021-11-09 08:03:21 +00:00
if (!user.getSubscribed().contains(channelId))
user.getSubscribed().add(channelId);
Use Hibernate as a database (#32) * Basic hibernate structure. * Commit stash. * Improve a lot of things. * Implement Feed, PubSub, query optimizations. * Update video views on channel visits. * Allow mass importing subscriptions. * Allow configuring database in config.properties. * Major multi-threading and pubsub improvements. * PubSub query improvements and fix for postgres. * Revert docker-compose change. * Disable showing sql statements.
2021-07-16 22:40:46 +00:00
Revert "Fix LBRY timeouts." This reverts commit a7b02e9566e72f6ea67169fc618e9dbf0483e002.
2021-11-09 08:03:21 +00:00
if (channelIds.length > 0) {
s.update(user);
s.beginTransaction().commit();
Fix LBRY timeouts.
2021-11-09 08:03:21 +00:00
}
Revert "Fix LBRY timeouts." This reverts commit a7b02e9566e72f6ea67169fc618e9dbf0483e002.
2021-11-09 08:03:21 +00:00
s.close();
Use Hibernate as a database (#32) * Basic hibernate structure. * Commit stash. * Improve a lot of things. * Implement Feed, PubSub, query optimizations. * Update video views on channel visits. * Allow mass importing subscriptions. * Allow configuring database in config.properties. * Major multi-threading and pubsub improvements. * PubSub query improvements and fix for postgres. * Revert docker-compose change. * Disable showing sql statements.
2021-07-16 22:40:46 +00:00
});
for (String channelId : channelIds) {
Multithreading.runAsyncLimited(() -> {
try {
Session sess = DatabaseSessionFactory.createSession();
me.kavin.piped.utils.obj.db.Channel channel = DatabaseHelper.getChannelFromId(sess, channelId);
if (channel == null) {
ChannelInfo info = null;
try {
info = ChannelInfo.getInfo("https://youtube.com/channel/" + channelId);
} catch (Exception e) {
return;
}
channel = new me.kavin.piped.utils.obj.db.Channel(channelId, info.getName(),
Implement uploaderVerified for feed response.
2021-08-17 17:29:15 +00:00
info.getAvatarUrl(), info.isVerified());
Use Hibernate as a database (#32) * Basic hibernate structure. * Commit stash. * Improve a lot of things. * Implement Feed, PubSub, query optimizations. * Update video views on channel visits. * Allow mass importing subscriptions. * Allow configuring database in config.properties. * Major multi-threading and pubsub improvements. * PubSub query improvements and fix for postgres. * Revert docker-compose change. * Disable showing sql statements.
2021-07-16 22:40:46 +00:00
sess.save(channel);
Multithreading.runAsync(() -> {
try {
Session sessSub = DatabaseSessionFactory.createSession();
subscribePubSub(channelId, sessSub);
sessSub.close();
Print stacktraces on any errors.
2021-07-21 12:37:43 +00:00
} catch (Exception e) {
Refractor and improve log handling.
2021-09-09 18:49:46 +00:00
ExceptionHandler.handle(e);
Use Hibernate as a database (#32) * Basic hibernate structure. * Commit stash. * Improve a lot of things. * Implement Feed, PubSub, query optimizations. * Update video views on channel visits. * Allow mass importing subscriptions. * Allow configuring database in config.properties. * Major multi-threading and pubsub improvements. * PubSub query improvements and fix for postgres. * Revert docker-compose change. * Disable showing sql statements.
2021-07-16 22:40:46 +00:00
}
});
for (StreamInfoItem item : info.getRelatedItems()) {
long time = item.getUploadDate() != null
? item.getUploadDate().offsetDateTime().toInstant().toEpochMilli()
: System.currentTimeMillis();
if ((System.currentTimeMillis() - time) < TimeUnit.DAYS.toMillis(10))
handleNewVideo(item.getUrl(), time, channel, sess);
}
if (!sess.getTransaction().isActive())
sess.getTransaction().begin();
sess.getTransaction().commit();
}
sess.close();
} catch (Exception e) {
Refractor and improve log handling.
2021-09-09 18:49:46 +00:00
ExceptionHandler.handle(e);
Use Hibernate as a database (#32) * Basic hibernate structure. * Commit stash. * Improve a lot of things. * Implement Feed, PubSub, query optimizations. * Update video views on channel visits. * Allow mass importing subscriptions. * Allow configuring database in config.properties. * Major multi-threading and pubsub improvements. * PubSub query improvements and fix for postgres. * Revert docker-compose change. * Disable showing sql statements.
2021-07-16 22:40:46 +00:00
}
});
}
return Constants.mapper.writeValueAsBytes(new AcceptedResponse());
}
s.close();
return Constants.mapper.writeValueAsBytes(new AuthenticationFailureResponse());
add playlists and small changes.
2021-01-04 05:47:28 +00:00
}
Add route to see subscriptions.
2021-07-22 20:04:21 +00:00
public static final byte[] subscriptionsResponse(String session)
throws IOException, NoSuchAlgorithmException, InvalidKeySpecException {
Session s = DatabaseSessionFactory.createSession();
User user = DatabaseHelper.getUserFromSessionWithSubscribed(s, session);
if (user != null) {
List<SubscriptionChannel> subscriptionItems = new ObjectArrayList<>();
if (user.getSubscribed() != null && !user.getSubscribed().isEmpty()) {
List<me.kavin.piped.utils.obj.db.Channel> channels = DatabaseHelper.getChannelFromIds(s,
user.getSubscribed());
channels.forEach(channel -> {
subscriptionItems.add(new SubscriptionChannel("/channel/" + channel.getUploaderId(),
channel.getUploader(), rewriteURL(channel.getUploaderAvatar()), channel.isVerified()));
});
Collections.sort(subscriptionItems, (a, b) -> (a.name.compareTo(b.name)));
}
s.close();
return Constants.mapper.writeValueAsBytes(subscriptionItems);
}
s.close();
return Constants.mapper.writeValueAsBytes(new AuthenticationFailureResponse());
}
Add registered users badge.
2021-11-11 11:35:50 +00:00
public static final String registeredBadgeRedirect() {
Session s = DatabaseSessionFactory.createSession();
long registered = ((Long) s.createQuery("select count(*) from User").uniqueResult()).longValue();
s.close();
return String.format("https://img.shields.io/badge/Registered%%20Users-%s-blue", String.valueOf(registered));
}
Add lbryId to streams response.
2021-08-08 11:17:58 +00:00
private static final String getLBRYId(String videoId) throws IOException, InterruptedException {
return new JSONObject(Constants.h2client.send(HttpRequest
Commit changes.
2021-02-24 09:52:29 +00:00
.newBuilder(URI.create("https://api.lbry.com/yt/resolve?video_ids=" + URLUtils.silentEncode(videoId)))
.setHeader("User-Agent", Constants.USER_AGENT).build(), BodyHandlers.ofString()).body())
.getJSONObject("data").getJSONObject("videos").optString(videoId);
Add lbryId to streams response.
2021-08-08 11:17:58 +00:00
}
Fix LBRY timeouts.
2021-11-09 08:06:13 +00:00
private static final String getLBRYStreamURL(String lbryId)
Add lbryId to streams response.
2021-08-08 11:17:58 +00:00
throws IOException, InterruptedException, ExecutionException {
Fix LBRY timeouts.
2021-11-09 08:06:13 +00:00
if (lbryId != null && !lbryId.isEmpty())
Don't fail hard when LBRY throws errors.
2021-10-06 15:37:30 +00:00
return new JSONObject(
Constants.h2client.send(
HttpRequest.newBuilder(URI.create("https://api.lbry.tv/api/v1/proxy?m=get"))
.POST(BodyPublishers.ofString(
String.valueOf(new JSONObject().put("id", System.currentTimeMillis())
.put("jsonrpc", "2.0").put("method", "get").put("params",
new JSONObject().put("uri", "lbry://" + lbryId)
.put("save_file", true)))))
.build(),
BodyHandlers.ofString()).body()).getJSONObject("result").getString("streaming_url");
Commit changes.
2021-02-24 09:52:29 +00:00
return null;
Initial commit.
2020-11-12 21:19:45 +00:00
}
Use Hibernate as a database (#32) * Basic hibernate structure. * Commit stash. * Improve a lot of things. * Implement Feed, PubSub, query optimizations. * Update video views on channel visits. * Allow mass importing subscriptions. * Allow configuring database in config.properties. * Major multi-threading and pubsub improvements. * PubSub query improvements and fix for postgres. * Revert docker-compose change. * Disable showing sql statements.
2021-07-16 22:40:46 +00:00
public static void handleNewVideo(String url, long time, me.kavin.piped.utils.obj.db.Channel channel, Session s) {
try {
handleNewVideo(StreamInfo.getInfo(url), time, channel, s);
Fix feed updates on manual subscription.
2021-07-17 14:02:57 +00:00
} catch (Exception e) {
Refractor and improve log handling.
2021-09-09 18:49:46 +00:00
ExceptionHandler.handle(e);
Use Hibernate as a database (#32) * Basic hibernate structure. * Commit stash. * Improve a lot of things. * Implement Feed, PubSub, query optimizations. * Update video views on channel visits. * Allow mass importing subscriptions. * Allow configuring database in config.properties. * Major multi-threading and pubsub improvements. * PubSub query improvements and fix for postgres. * Revert docker-compose change. * Disable showing sql statements.
2021-07-16 22:40:46 +00:00
}
}
private static void handleNewVideo(StreamInfo info, long time, me.kavin.piped.utils.obj.db.Channel channel,
Session s) {
if (channel == null)
channel = DatabaseHelper.getChannelFromId(s,
info.getUploaderUrl().substring("https://www.youtube.com/channel/".length()));
long infoTime = info.getUploadDate() != null ? info.getUploadDate().offsetDateTime().toInstant().toEpochMilli()
: System.currentTimeMillis();
Video video = null;
if (channel != null && (video = DatabaseHelper.getVideoFromId(s, info.getId())) == null
&& (System.currentTimeMillis() - infoTime) < TimeUnit.DAYS.toMillis(10)) {
video = new Video(info.getId(), info.getName(), info.getViewCount(), info.getDuration(),
Math.max(infoTime, time), info.getThumbnailUrl(), channel);
s.save(video);
if (!s.getTransaction().isActive())
s.getTransaction().begin();
s.getTransaction().commit();
} else if (video != null) {
video.setViews(info.getViewCount());
s.update(video);
Fix "Transaction not successfully started".
2021-07-20 20:57:02 +00:00
if (!s.getTransaction().isActive())
s.getTransaction().begin();
Use Hibernate as a database (#32) * Basic hibernate structure. * Commit stash. * Improve a lot of things. * Implement Feed, PubSub, query optimizations. * Update video views on channel visits. * Allow mass importing subscriptions. * Allow configuring database in config.properties. * Major multi-threading and pubsub improvements. * PubSub query improvements and fix for postgres. * Revert docker-compose change. * Disable showing sql statements.
2021-07-16 22:40:46 +00:00
s.getTransaction().commit();
}
}
private static void updateViews(String id, long views, long time, boolean addIfNonExistent) {
Multithreading.runAsync(() -> {
try {
Session s = DatabaseSessionFactory.createSession();
Video video = DatabaseHelper.getVideoFromId(s, id);
if (video != null) {
video.setViews(views);
s.update(video);
s.beginTransaction().commit();
} else if (addIfNonExistent)
handleNewVideo("https://www.youtube.com/watch?v=" + id, time, null, s);
s.close();
} catch (Exception e) {
Refractor and improve log handling.
2021-09-09 18:49:46 +00:00
ExceptionHandler.handle(e);
Use Hibernate as a database (#32) * Basic hibernate structure. * Commit stash. * Improve a lot of things. * Implement Feed, PubSub, query optimizations. * Update video views on channel visits. * Allow mass importing subscriptions. * Allow configuring database in config.properties. * Major multi-threading and pubsub improvements. * PubSub query improvements and fix for postgres. * Revert docker-compose change. * Disable showing sql statements.
2021-07-16 22:40:46 +00:00
}
});
}
public static void subscribePubSub(String channelId, Session s) throws IOException, InterruptedException {
PubSub pubsub = DatabaseHelper.getPubSubFromId(s, channelId);
if (pubsub == null || System.currentTimeMillis() - pubsub.getSubbedAt() > TimeUnit.DAYS.toMillis(4)) {
String callback = Constants.PUBLIC_URL + "/webhooks/pubsub";
String topic = "https://www.youtube.com/xml/feeds/videos.xml?channel_id=" + channelId;
Builder builder = HttpRequest.newBuilder(URI.create("https://pubsubhubbub.appspot.com/subscribe"));
Map<String, String> formParams = new Object2ObjectOpenHashMap<>();
StringBuilder formBody = new StringBuilder();
builder.header("content-type", "application/x-www-form-urlencoded");
formParams.put("hub.callback", callback);
formParams.put("hub.topic", topic);
formParams.put("hub.verify", "async");
formParams.put("hub.mode", "subscribe");
formParams.put("hub.lease_seconds", "432000");
formParams.forEach((name, value) -> {
formBody.append(name + "=" + URLUtils.silentEncode(value) + "&");
});
builder.method("POST",
BodyPublishers.ofString(String.valueOf(formBody.substring(0, formBody.length() - 1))));
Log pubsub failures.
2021-07-22 09:43:27 +00:00
HttpResponse<InputStream> resp = Constants.h2client.send(builder.build(), BodyHandlers.ofInputStream());
Use Hibernate as a database (#32) * Basic hibernate structure. * Commit stash. * Improve a lot of things. * Implement Feed, PubSub, query optimizations. * Update video views on channel visits. * Allow mass importing subscriptions. * Allow configuring database in config.properties. * Major multi-threading and pubsub improvements. * PubSub query improvements and fix for postgres. * Revert docker-compose change. * Disable showing sql statements.
2021-07-16 22:40:46 +00:00
Resp code should be 202 and not 204.
2021-07-22 09:59:59 +00:00
if (resp.statusCode() == 202) {
Log pubsub failures.
2021-07-22 09:43:27 +00:00
if (pubsub == null)
pubsub = new PubSub(channelId, System.currentTimeMillis());
else
pubsub.setSubbedAt(System.currentTimeMillis());
Use Hibernate as a database (#32) * Basic hibernate structure. * Commit stash. * Improve a lot of things. * Implement Feed, PubSub, query optimizations. * Update video views on channel visits. * Allow mass importing subscriptions. * Allow configuring database in config.properties. * Major multi-threading and pubsub improvements. * PubSub query improvements and fix for postgres. * Revert docker-compose change. * Disable showing sql statements.
2021-07-16 22:40:46 +00:00
Log pubsub failures.
2021-07-22 09:43:27 +00:00
s.saveOrUpdate(pubsub);
Use Hibernate as a database (#32) * Basic hibernate structure. * Commit stash. * Improve a lot of things. * Implement Feed, PubSub, query optimizations. * Update video views on channel visits. * Allow mass importing subscriptions. * Allow configuring database in config.properties. * Major multi-threading and pubsub improvements. * PubSub query improvements and fix for postgres. * Revert docker-compose change. * Disable showing sql statements.
2021-07-16 22:40:46 +00:00
Log pubsub failures.
2021-07-22 09:43:27 +00:00
if (!s.getTransaction().isActive())
s.getTransaction().begin();
s.getTransaction().commit();
} else
System.out.println(
"Failed to subscribe: " + resp.statusCode() + "\n" + IOUtils.toString(resp.body(), "UTF-8"));
Use Hibernate as a database (#32) * Basic hibernate structure. * Commit stash. * Improve a lot of things. * Implement Feed, PubSub, query optimizations. * Update video views on channel visits. * Allow mass importing subscriptions. * Allow configuring database in config.properties. * Major multi-threading and pubsub improvements. * PubSub query improvements and fix for postgres. * Revert docker-compose change. * Disable showing sql statements.
2021-07-16 22:40:46 +00:00
}
}
Use substringAfter to cleanup code and fix YouTube music URLs.
2021-09-05 18:44:24 +00:00
private static final String substringYouTube(String s) {
return s == null || s.isEmpty() ? null : StringUtils.substringAfter(s, "youtube.com");
Add support for search filters and simplify continuations.
2021-06-14 19:44:23 +00:00
}
Refractor repeated code.
2021-09-09 19:25:16 +00:00
private static StreamItem collectRelatedStream(Object o) {
StreamInfoItem item = (StreamInfoItem) o;
return new StreamItem(substringYouTube(item.getUrl()), item.getName(), rewriteURL(item.getThumbnailUrl()),
item.getUploaderName(), substringYouTube(item.getUploaderUrl()),
rewriteURL(item.getUploaderAvatarUrl()), item.getTextualUploadDate(), item.getDuration(),
item.getViewCount(), item.isUploaderVerified());
}
Initial commit.
2020-11-12 21:19:45 +00:00
private static String rewriteURL(final String old) {
Commit changes.
2021-02-24 09:52:29 +00:00
if (old == null || old.isEmpty())
return null;
Better proxying of images.
2020-12-09 13:29:12 +00:00
Commit changes.
2021-02-24 09:52:29 +00:00
URL url = null;
Initial commit.
2020-11-12 21:19:45 +00:00
Commit changes.
2021-02-24 09:52:29 +00:00
try {
url = new URL(old);
} catch (MalformedURLException e) {
Refractor and improve log handling.
2021-09-09 18:49:46 +00:00
ExceptionHandler.handle(e);
Commit changes.
2021-02-24 09:52:29 +00:00
}
Initial commit.
2020-11-12 21:19:45 +00:00
Commit changes.
2021-02-24 09:52:29 +00:00
final String host = url.getHost();
Initial commit.
2020-11-12 21:19:45 +00:00
Commit changes.
2021-02-24 09:52:29 +00:00
String query = url.getQuery();
Initial commit.
2020-11-12 21:19:45 +00:00
Commit changes.
2021-02-24 09:52:29 +00:00
boolean hasQuery = query != null;
Initial commit.
2020-11-12 21:19:45 +00:00
Commit changes.
2021-02-24 09:52:29 +00:00
String path = url.getPath();
Initial commit.
2020-11-12 21:19:45 +00:00
Fix url rewrite issues Closes https://github.com/TeamPiped/Piped/issues/598
2021-11-04 17:45:33 +00:00
if (path.contains("=")) {
path = StringUtils.substringBefore(path, "=") + "="
+ StringUtils.substringAfter(path, "=").replace("-rj", "-rw");
}
Initial commit.
2020-11-12 21:19:45 +00:00
Commit changes.
2021-02-24 09:52:29 +00:00
return Constants.PROXY_PART + path + (hasQuery ? "?" + query + "&host=" : "?host=")
+ URLUtils.silentEncode(host);
Initial commit.
2020-11-12 21:19:45 +00:00
}
}
Reference in a new issue Copy permalink