tdlight-team/tdlight
tdlight-team/tdlight
1
1
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Remove most memory related features

Browse Source 
I can't maintain anymore this amount of features while keeping the library constantly updated and without bugs. Every merge was taking me multiple hours of revisioning the code. I give up.
From this commit onwards TDLight will only have small useful customizations that are easy to maintain.
Now the people relying on the OptimizeMemory method can restart the session every N hours to free up the memory.
The real way to keep a low memory usage must involve a huge refactoring to allow the unloading of the caches into the sqlite database, similar to what's already happening with messages data. Only Levlam has the ability to implement this without needing to merge the upstream everytime.
This commit is contained in:
Andrea Cavalli 2021-09-25 22:11:42 +02:00
parent 897d4af16c
commit 45e855f89d
70 changed files with 473 additions and 2342 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
CHANGELOG.md
View File

@ -622,7 +622,7 @@ Changes in 1.5.0 (9 Sep 2019):
* Added the class `chatEventPollStopped` representing the closing of a poll in a message in the chat event log.
* Added ability to specify the exact types of problems with a call in the method `sendCallRating` and
the new class `CallProblem`.
* Changes in [tdweb](https://github.com/tdlib/td/blob/master/example/web/):
* Changes in [tdweb](https://github.com/tdlight-team/tdlight/blob/master/example/web/):
- Supported non-zero `offset` and `limit` in `readFilePart`.
-----------------------------------------------------------------------------------------------------------------------
@ -632,7 +632,7 @@ Changes in 1.4.0 (1 May 2019):
* Added a [TDLib build instructions generator](https://tdlib.github.io/td/build.html), covering in details
TDLib building on the most popular operating systems.
* Added an example of TDLib building and usage from a browser.
See https://github.com/tdlib/td/blob/master/example/web/ for more details.
See https://github.com/tdlight-team/tdlight/blob/master/example/web/ for more details.
* Allowed to pass NULL pointer to `td_json_client_execute` instead of a previously created JSON client.
Now you can use synchronous TDLib methods through a JSON interface before creating a TDLib JSON client.
* Added support for media streaming by allowing to download any part of a file:
@ -883,7 +883,7 @@ Changes in 1.4.0 (1 May 2019):
Changes in 1.3.0 (5 Sep 2018):
* Added a review of existing TDLib based [frameworks](https://github.com/tdlib/td/blob/master/example/README.md)
* Added a review of existing TDLib based [frameworks](https://github.com/tdlight-team/tdlight/blob/master/example/README.md)
in different programming languages.
* Added a [Getting started](https://core.telegram.org/tdlib/getting-started) guide describing the main TDLib concepts
and basic principles required for library usage.

120
README.md
View File

@ -1,82 +1,12 @@
# TDLight
TDLight is a fork of tdlib, focused on memory footprint and performance.
TDLight is 100% compatible with tdlib, if you don't use the sqlite database.
⚠️ Memory Cleanup remove nearly every cached value, so you must use it with caution!
This function is not suggested for GUI clients, because they heavily rely on retrieving cached data!
TDLib developers strongly advise against the use of this feature, since it is not an intended behavior.
## Added features
### Memory cleanup
TDLight can clean itself and release some ram to the OS if you want. Look at **TdApi.OptimizeMemory** in "Modified features" paragraph to see how.
### Constant memory usage without restarting
TDLight, if used with care, doesn't grow in memory usage with time. Look at **TdApi.OptimizeMemory** in "Modified features" paragraph to see how
![memory usage](info/memory-usage.jpg)
### Custom options
We added some options:
* **disable_minithumbnails** (true/**false**) This setting removes minithumbnails everywhere. It reduces memory usage because tdlib keeps them in RAM
* **disable_document_filenames** (true/**false**) If you don't care about having the original filenames of every file stored in RAM, you can disable them using this option. It reduces memory usage
* **disable_notifications** (true/**false**) In TDLib pending notification updates are stored in ram until you "read" them. This option disables completely notifications and keeps the pending notifications queue empty, reducing memory usage
* **ignore_update_chat_last_message** (true/**false**) If you don't care about have updateChatLastMessage updates enable this
* **ignore_update_chat_read_inbox** (true/**false**) If you don't care about have updateChatReadInbox updates enable this
* **ignore_update_user_chat_action** (true/**false**) If you don't care about have updateUserChatAction updates enable this
* **ignore_server_deletes_and_reads** (true/**false**) If you don't care about receiving read receipts and remote deletes from other users, enable this, it will reduce memory usage
* **delete_chat_reference_after_seconds** (positive number) During cleanup, free the memory of the chats that have not been touched for more than X seconds
* **delete_user_reference_after_seconds** (positive number) During cleanup, free the memory of the users that have not been touched for more than X seconds
* **delete_file_reference_after_seconds** (positive number) During cleanup, free the memory of the files that have not been touched for more than X seconds
* **receive_access_hashes** (true/**false**) Receive chats and users access hash as updates
* **experiment_enable_file_reference_cleanup** (**true**/false) During cleanup, free the memory of the file references
* **experiment_enable_chat_access_hash_cleanup** (**true**/false) During cleanup, clean chats and channels access hash
* **enable_reactive_channel_difference** (true/**false**) Enable manual `get_channel_difference` execution by calling `getChannelDifference(channel_difference_id)`.
Don't modify this option unless you have a very large bot that struggles to keep up with start-up updates throughput, or you want to use a reactive library.
## Custom API functions
### TdApi.OptimizeMemory
This method is used to optimize the memory usage, but it must be used carefully.
It removes almost all cached values and releases the memory back to the OS.
You can call TdApi.OptimizeMemory normally, but removing cached values
can cause problems if you don't take some precautions.
If you want to avoid receiving data with missing fields during cleanup:
1. Before calling *TdApi.OptimizeMemory* you must:
1. Read all the pending updates to empty the pending updates queue.
2. Disable internet connection using *TdApi.SetNetworkType(TdApi.NetworkTypeNone)*
2. Call *TdApi.OptimizeMemory*
3. After calling *TdApi.OptimizeMemory* you must:
1. **NOT** use the old file ids because they have been deleted! (Example: If you receive the file 12 after OptimizeMemory is not the same file 12 that you received before *TdApi.OptimizeMemory*, because the id 12 has been reused)
2. Re-enable internet connection using *TdApi.SetNetworkType(TdApi.NetworkTypeOther)*
### TdApi.GetMemoryStatistics
This method is used to read the size of all the biggest data maps inside tdlib implementation.
The output contains a string that can be parsed as a JSON.
## Other recommended options
* Options:
* ignore_inline_thumbnails: true
* disable_top_chats: true
* ignore_platform_restrictions: true
* ignore_sensitive_content_restrictions: true
* Disable all the databases (messages_db, users_db, files_db)
⚠️ If you use the databases, TDLight memory cleanup feature will be automatically disabled, because databases will lose some data when cleaning up the memory.
-----
The following text is the classic TDLib readme:
# TDLib
TDLib (Telegram Database library) is a cross-platform library for building [Telegram](https://telegram.org) clients. It can be easily used from almost any programming language.
TDLight is a fork of TDLib, a cross-platform library for building [Telegram](https://telegram.org) clients. It can be easily used from almost any programming language.
## Table of Contents
- [Features](#features)
- [TDLight extra features](#tdlight-extra-features)
- [TDLight extra API functions](#tdlight-extra-api-functions)
- [TDLight recommended options](#tdlight-recommended-options)
- [Examples and documentation](#usage)
- [Dependencies](#dependencies)
- [Building](#building)
@ -101,17 +31,41 @@ TDLib (Telegram Database library) is a cross-platform library for building [Tele
* **Secure**: all local data is encrypted using a user-provided encryption key.
* **Fully-asynchronous**: requests to `TDLib` don't block each other or anything else, responses are sent when they are available.
<a name="tdlight-extra-features"></a>
### TDLight extra features
#### TDLight extra options
* **disable_minithumbnails** (true/**false**) This setting removes minithumbnails everywhere. It reduces memory usage because tdlib keeps them in RAM
* **disable_document_filenames** (true/**false**) If you don't care about having the original filenames of every file stored in RAM, you can disable them using this option. It reduces memory usage
* **disable_notifications** (true/**false**) In TDLib pending notification updates are stored in ram until you "read" them. This option disables completely notifications and keeps the pending notifications queue empty, reducing memory usage
* **ignore_update_chat_last_message** (true/**false**) If you don't care about have updateChatLastMessage updates enable this
* **ignore_update_chat_read_inbox** (true/**false**) If you don't care about have updateChatReadInbox updates enable this
* **ignore_update_user_chat_action** (true/**false**) If you don't care about have updateUserChatAction updates enable this
* **ignore_server_deletes_and_reads** (true/**false**) If you don't care about receiving read receipts and remote deletes from other users, enable this, it will reduce memory usage
* **receive_access_hashes** (true/**false**) Receive chats and users access hash as updates
<a name="tdlight-extra-api-functions"></a>
### TDLight extra API functions
#### TdApi.GetMemoryStatistics
This method is used to read the size of all the internal TDLib data structures.
The output contains a string that can be parsed as a JSON.
<a name="tdlight-recommended-options"></a>
## TDLight recommended options
* Options:
* ignore_inline_thumbnails: true
* disable_top_chats: true
* ignore_platform_restrictions: true
* ignore_sensitive_content_restrictions: true
* Disable all the databases (messages_db, users_db, files_db)
<a name="usage"></a>
## Examples and documentation
See our [Getting Started](https://core.telegram.org/tdlib/getting-started) tutorial for a description of basic TDLib concepts.
Take a look at our [examples](https://github.com/tdlib/td/blob/master/example/README.md#tdlib-usage-and-build-examples).
Take a look at our [examples](https://github.com/tdlight-team/tdlight/blob/master/example/README.md#tdlib-usage-and-build-examples).
See a [TDLight build instructions generator](https://tdlight-team.github.io/tdlight/build.html) for detailed instructions on how to build TDLib.
See description of our [JSON](#using-json), [C++](#using-cxx), [Java](#using-java) and [.NET](#using-dotnet) interfaces.
See the [td_api.tl](https://github.com/tdlib/td/blob/master/td/generate/scheme/td_api.tl) scheme or the automatically generated [HTML documentation](https://core.telegram.org/tdlib/docs/td__api_8h.html)
See the [td_api.tl](https://github.com/tdlight-team/tdlight/blob/master/td/generate/scheme/td_api.tl) scheme or the automatically generated [HTML documentation](https://core.telegram.org/tdlib/docs/td__api_8h.html)
for a list of all available `TDLib` [methods](https://core.telegram.org/tdlib/docs/classtd_1_1td__api_1_1_function.html) and [classes](https://core.telegram.org/tdlib/docs/classtd_1_1td__api_1_1_object.html).
<a name="dependencies"></a>
@ -140,7 +94,7 @@ cmake -DCMAKE_BUILD_TYPE=Release ..
cmake --build .
```
To build `TDLib` on low memory devices you can run [SplitSource.php](https://github.com/tdlib/td/blob/master/SplitSource.php) script
To build `TDLib` on low memory devices you can run [SplitSource.php](https://github.com/tdlight-team/tdlight/blob/master/SplitSource.php) script
before compiling main `TDLib` source code and compile only needed targets:
```
mkdir build
@ -179,21 +133,21 @@ Or you could install `TDLib` and then reference it in your CMakeLists.txt like t
find_package(Td 1.7.8 REQUIRED)
target_link_libraries(YourTarget PRIVATE Td::TdStatic)
```
See [example/cpp/CMakeLists.txt](https://github.com/tdlib/td/tree/master/example/cpp/CMakeLists.txt).
See [example/cpp/CMakeLists.txt](https://github.com/tdlight-team/tdlight/tree/master/example/cpp/CMakeLists.txt).
<a name="using-java"></a>
## Using in Java projects
`TDLib` provides native Java interface through JNI. To enable it, specify option `-DTD_ENABLE_JNI=ON` to CMake.
See [example/java](https://github.com/tdlib/td/tree/master/example/java) for example of using `TDLib` from Java and detailed build and usage instructions.
See [example/java](https://github.com/tdlight-team/tdlight/tree/master/example/java) for example of using `TDLib` from Java and detailed build and usage instructions.
<a name="using-dotnet"></a>
## Using in .NET projects
`TDLib` provides native .NET interface through `C++/CLI` and `C++/CX`. To enable it, specify option `-DTD_ENABLE_DOTNET=ON` to CMake.
.NET Core supports `C++/CLI` only since version 3.1 and only on Windows, so if older .NET Core is used or portability is needed, then `TDLib` JSON interface should be used through P/Invoke instead.
See [example/csharp](https://github.com/tdlib/td/tree/master/example/csharp) for example of using `TDLib` from C# and detailed build and usage instructions.
See [example/uwp](https://github.com/tdlib/td/tree/master/example/uwp) for example of using `TDLib` from C# UWP application and detailed build and usage instructions for Visual Studio Extension "TDLib for Universal Windows Platform".
See [example/csharp](https://github.com/tdlight-team/tdlight/tree/master/example/csharp) for example of using `TDLib` from C# and detailed build and usage instructions.
See [example/uwp](https://github.com/tdlight-team/tdlight/tree/master/example/uwp) for example of using `TDLib` from C# UWP application and detailed build and usage instructions for Visual Studio Extension "TDLib for Universal Windows Platform".
When `TDLib` is built with `TD_ENABLE_DOTNET` option enabled, `C++` documentation is removed from some files. You need to checkout these files to return `C++` documentation back:
```
@ -205,13 +159,13 @@ git checkout td/telegram/Client.h td/telegram/Log.h td/tl/TlObject.h
`TDLib` provides efficient native C++, Java, and .NET interfaces.
But for most use cases we suggest to use the JSON interface, which can be easily used with any programming language that is able to execute C functions.
See [td_json_client](https://core.telegram.org/tdlib/docs/td__json__client_8h.html) documentation for detailed JSON interface description,
the [td_api.tl](https://github.com/tdlib/td/blob/master/td/generate/scheme/td_api.tl) scheme or the automatically generated [HTML documentation](https://core.telegram.org/tdlib/docs/td__api_8h.html) for a list of
the [td_api.tl](https://github.com/tdlight-team/tdlight/blob/master/td/generate/scheme/td_api.tl) scheme or the automatically generated [HTML documentation](https://core.telegram.org/tdlib/docs/td__api_8h.html) for a list of
all available `TDLib` [methods](https://core.telegram.org/tdlib/docs/classtd_1_1td__api_1_1_function.html) and [classes](https://core.telegram.org/tdlib/docs/classtd_1_1td__api_1_1_object.html).
`TDLib` JSON interface adheres to semantic versioning and versions with the same major version number are binary and backward compatible, but the underlying `TDLib` API can be different for different minor and even patch versions.
If you need to support different `TDLib` versions, then you can use a value of the `version` option to find exact `TDLib` version to use appropriate API methods.
See [example/python/tdjson_example.py](https://github.com/tdlib/td/tree/master/example/python/tdjson_example.py) for an example of such usage.
See [example/python/tdjson_example.py](https://github.com/tdlight-team/tdlight/tree/master/example/python/tdjson_example.py) for an example of such usage.
<a name="license"></a>
## License

2
build.html
View File

@ -544,7 +544,7 @@ function onOptionsChanged() {
}
if (os_mac) {
pre_text.push('Note that the following instruction will build TDLib only for the current architecture (x64 or Apple silicon).');
pre_text.push('If you want to create a universal XCFramework, take a look at our <a href="https://github.com/tdlib/td/tree/master/example/ios">example</a> instead.');
pre_text.push('If you want to create a universal XCFramework, take a look at our <a href="https://github.com/tdlight-team/tdlight/tree/master/example/ios">example</a> instead.');
}
var terminal_name = (function () {

26
example/README.md
View File

@ -1,7 +1,7 @@
# TDLib usage and build examples
This directory contains basic examples of TDLib usage from different programming languages and examples of library building for different platforms.
If you are looking for documentation of all available TDLib methods, see the [td_api.tl](https://github.com/tdlib/td/blob/master/td/generate/scheme/td_api.tl) scheme or the
If you are looking for documentation of all available TDLib methods, see the [td_api.tl](https://github.com/tdlight-team/tdlight/blob/master/td/generate/scheme/td_api.tl) scheme or the
automatically generated [HTML documentation](https://core.telegram.org/tdlib/docs/td__api_8h.html) for a list of all available TDLib
[methods](https://core.telegram.org/tdlib/docs/classtd_1_1td__api_1_1_function.html) and [classes](https://core.telegram.org/tdlib/docs/classtd_1_1td__api_1_1_object.html).
Also take a look at our [Getting Started](https://core.telegram.org/tdlib/getting-started) tutorial for a description of basic TDLib concepts.
@ -53,13 +53,13 @@ This wrapper contains automatically generated fully-documented classes for all T
For older Python versions you can use [pytdlib](https://github.com/pytdlib/pytdlib).
This wrapper contains generator for TDLib API classes and basic interface for interaction with TDLib.
You can also check out [example/python/tdjson_example.py](https://github.com/tdlib/td/tree/master/example/python/tdjson_example.py) and
You can also check out [example/python/tdjson_example.py](https://github.com/tdlight-team/tdlight/tree/master/example/python/tdjson_example.py) and
[tdlib-python](https://github.com/JunaidBabu/tdlib-python) for some very basic examples of TDLib JSON interface integration with Python.
<a name="javascript"></a>
## Using TDLib in JavaScript projects
TDLib can be compiled to WebAssembly or asm.js and used in a browser from JavaScript. See [tdweb](https://github.com/tdlib/td/tree/master/example/web) as a convenient wrapper for TDLib in a browser
TDLib can be compiled to WebAssembly or asm.js and used in a browser from JavaScript. See [tdweb](https://github.com/tdlight-team/tdlight/tree/master/example/web) as a convenient wrapper for TDLib in a browser
and [telegram-react](https://github.com/evgeny-nadymov/telegram-react) as an example of a TDLib-based Telegram client.
TDLib can be used from Node.js through the [JSON](https://github.com/tdlib/td#using-json) interface.
@ -90,7 +90,7 @@ You can also see [go-tdjson](https://github.com/L11R/go-tdjson) for another exam
TDLib can be used from the Java programming language through native [JNI](https://github.com/tdlib/td#using-java) binding.
We provide a generator for JNI bridge methods and Java classes for all TDLib API methods and objects.
See [example/java](https://github.com/tdlib/td/tree/master/example/java) for an example of using TDLib from desktop Java along with detailed building and usage instructions.
See [example/java](https://github.com/tdlight-team/tdlight/tree/master/example/java) for an example of using TDLib from desktop Java along with detailed building and usage instructions.
To use TDLib to create Android Java applications, use our [prebuilt library for Android](https://core.telegram.org/tdlib/tdlib.zip).
You can also see [JTDLib](https://github.com/ErnyTech/JTDLib) for another example of Java wrapper for TDLib.
@ -110,8 +110,8 @@ See also [KtLib](https://github.com/nekohasekai/KtLib), which provides an automa
TDLib provides a native [.NET](https://github.com/tdlib/td#using-dotnet) interface through `C++/CLI` and `C++/CX`.
See [tdlib-netcore](https://github.com/dantmnf/tdlib-netcore) for a SWIG-like binding with automatically generated classes for TDLib API.
See [example/uwp](https://github.com/tdlib/td/tree/master/example/uwp) for an example of building TDLib SDK for the Universal Windows Platform and an example of its usage from C#.
See [example/csharp](https://github.com/tdlib/td/tree/master/example/csharp) for an example of building TDLib with `C++/CLI` support and an example of TDLib usage from C# on Windows.
See [example/uwp](https://github.com/tdlight-team/tdlight/tree/master/example/uwp) for an example of building TDLib SDK for the Universal Windows Platform and an example of its usage from C#.
See [example/csharp](https://github.com/tdlight-team/tdlight/tree/master/example/csharp) for an example of building TDLib with `C++/CLI` support and an example of TDLib usage from C# on Windows.
If you want to write a cross-platform C# application using .NET Core, see [tdsharp](https://github.com/egramtel/tdsharp). It uses our [JSON](https://github.com/tdlib/td#using-json) interface,
provides an asynchronous interface for interaction with TDLib, automatically generated classes for TDLib API and has some examples.
@ -125,8 +125,8 @@ Also see [Unigram](https://github.com/UnigramDev/Unigram), which is a full-featu
TDLib has a simple and convenient C++11-interface for sending and receiving requests and can be statically linked to your application.
See [example/cpp](https://github.com/tdlib/td/tree/master/example/cpp) for an example of TDLib usage from C++.
[td_example.cpp](https://github.com/tdlib/td/tree/master/example/cpp/td_example.cpp) contains an example of authorization, processing new incoming messages, getting a list of chats and sending a text message.
See [example/cpp](https://github.com/tdlight-team/tdlight/tree/master/example/cpp) for an example of TDLib usage from C++.
[td_example.cpp](https://github.com/tdlight-team/tdlight/tree/master/example/cpp/td_example.cpp) contains an example of authorization, processing new incoming messages, getting a list of chats and sending a text message.
See also the source code of [Fernschreiber](https://github.com/Wunderfitz/harbour-fernschreiber) and [Depecher](https://github.com/blacksailer/depecher) – Telegram apps for Sailfish OS,
[TELEports](https://gitlab.com/ubports/apps/teleports) – a Qt-client for Ubuntu Touch, or [tdlib-purple](https://github.com/ars3niy/tdlib-purple) - Telegram plugin for Pidgin, all of which are based on TDLib.
@ -136,18 +136,18 @@ See also the source code of [Fernschreiber](https://github.com/Wunderfitz/harbou
TDLib can be used from the Swift programming language through the [JSON](https://github.com/tdlib/td#using-json) interface and can be linked statically or dynamically.
See [example/ios](https://github.com/tdlib/td/tree/master/example/ios) for an example of building TDLib for iOS, watchOS, tvOS, and macOS.
See [example/ios](https://github.com/tdlight-team/tdlight/tree/master/example/ios) for an example of building TDLib for iOS, watchOS, tvOS, and macOS.
See [TDLibKit](https://github.com/Swiftgram/TDLibKit), [tdlib-swift](https://github.com/modestman/tdlib-swift), or [TDLib-iOS](https://github.com/leoMehlig/TDLib-iOS), which provide convenient TDLib clients with automatically generated and fully-documented classes for all TDLib API methods and objects.
See [example/swift](https://github.com/tdlib/td/tree/master/example/swift) for an example of a macOS Swift application.
See [example/swift](https://github.com/tdlight-team/tdlight/tree/master/example/swift) for an example of a macOS Swift application.
<a name="objective-c"></a>
## Using TDLib in Objective-C projects
TDLib can be used from the Objective-C programming language through [JSON](https://github.com/tdlib/td#using-json) interface and can be linked statically or dynamically.
See [example/ios](https://github.com/tdlib/td/tree/master/example/ios) for an example of building TDLib for iOS, watchOS, tvOS, and macOS.
See [example/ios](https://github.com/tdlight-team/tdlight/tree/master/example/ios) for an example of building TDLib for iOS, watchOS, tvOS, and macOS.
<a name="object-pascal"></a>
## Using TDLib in Object Pascal projects with Delphi and Lazarus
@ -163,7 +163,7 @@ See [tdlib-lazarus](https://github.com/dieletro/tdlib-lazarus) for an example of
TDLib can be used from the Dart programming language through the [JSON](https://github.com/tdlib/td#using-json) interface and a Dart Native Extension or Dart FFI.
See [dart_tdlib](https://github.com/periodicaidan/dart_tdlib) or [Dart wrapper for TDLib](https://github.com/tdlib/td/pull/708/commits/237060abd4c205768153180e9f814298d1aa9d49) for an example of a TDLib Dart bindings through FFI.
See [dart_tdlib](https://github.com/periodicaidan/dart_tdlib) or [Dart wrapper for TDLib](https://github.com/tdlight-team/tdlight/pull/708/commits/237060abd4c205768153180e9f814298d1aa9d49) for an example of a TDLib Dart bindings through FFI.
See [project.scarlet](https://github.com/aaugmentum/project.scarlet), [tdlib](https://github.com/i-Naji/tdlib), [tdlib](https://github.com/pqhaz/tdlib),
[tdlib-dart](https://github.com/drewpayment/tdlib-dart), [tdlib-dart](https://github.com/triedcatched/tdlib-dart), or [telegram-service](https://github.com/igorder-dev/telegram-service) for examples of using TDLib from Dart.
@ -282,7 +282,7 @@ See [TDLib bindings for 1С:Enterprise](https://github.com/Infactum/telegram-nat
TDLib can be used from the C programming language through the [JSON](https://github.com/tdlib/td#using-json) interface and can be linked statically or dynamically.
You can also try to use our [C](https://github.com/tdlib/td/blob/master/td/telegram/td_c_client.h) client, which was used by the private TDLib-based version of [telegram-cli](https://github.com/vysheng/tg).
You can also try to use our [C](https://github.com/tdlight-team/tdlight/blob/master/td/telegram/td_c_client.h) client, which was used by the private TDLib-based version of [telegram-cli](https://github.com/vysheng/tg).
<a name="other"></a>
## Using TDLib from other programming languages

4
example/ios/README.md
View File

@ -4,7 +4,7 @@ Below are instructions for building TDLib for iOS, watchOS, tvOS, and also macOS
If you need only a macOS build for the current architecture, take a look at [TDLib build instructions generator](https://tdlib.github.io/td/build.html).
For example of usage take a look at our [Swift example](https://github.com/tdlib/td/tree/master/example/swift).
For example of usage take a look at our [Swift example](https://github.com/tdlight-team/tdlight/tree/master/example/swift).
To compile `TDLib` you will need to:
* Install the latest Xcode via `xcode-select --install` or downloading it from [Xcode website](https://developer.apple.com/xcode/).
@ -37,7 +37,7 @@ cd <path to TDLib sources>/example/ios
```
This may take a while, because TDLib will be built about 16 times.
Resulting XCFramework will work on any architecture and even on a simulator (x64, Apple silicon).
We use [CMake/iOS.cmake](https://github.com/tdlib/td/blob/master/CMake/iOS.cmake) toolchain, other toolchains may work too.
We use [CMake/iOS.cmake](https://github.com/tdlight-team/tdlight/blob/master/CMake/iOS.cmake) toolchain, other toolchains may work too.
Built libraries and XCFramework will be stored in `tdjson` directory.

2
example/uwp/extension.vsixmanifest
View File

@ -6,7 +6,7 @@
<MoreInfo>https://core.telegram.org/tdlib</MoreInfo>
<Tags>Telegram, TDLib, library, client, API</Tags>
<License>LICENSE_1_0.txt</License>
<ReleaseNotes>https://github.com/tdlib/td/blob/master/CHANGELOG.md</ReleaseNotes>
<ReleaseNotes>https://github.com/tdlight-team/tdlight/blob/master/CHANGELOG.md</ReleaseNotes>
</Metadata>
<Installation Scope="Global">
<InstallationTarget Id="Microsoft.ExtensionSDK" TargetPlatformIdentifier="UAP" TargetPlatformVersion="v0.8.0.0" SdkName="Telegram.Td.UWP" SdkVersion="1.0" />

2
example/web/README.md
View File

@ -26,4 +26,4 @@ cd <path to TDLib sources>/example/web
## Using tdweb NPM package
See [tdweb](https://www.npmjs.com/package/tdweb) or [README.md](https://github.com/tdlib/td/tree/master/example/web/tdweb/README.md) for package documentation.
See [tdweb](https://www.npmjs.com/package/tdweb) or [README.md](https://github.com/tdlight-team/tdlight/tree/master/example/web/tdweb/README.md) for package documentation.

4
example/web/tdweb/README.md
View File

@ -8,11 +8,11 @@ Once this is done, you can send queries to the TDLib instance using the method `
See [Getting Started](https://core.telegram.org/tdlib/getting-started) for a description of basic TDLib concepts and a short introduction to TDLib usage.
See the [td_api.tl](https://github.com/tdlib/td/blob/master/td/generate/scheme/td_api.tl) scheme or
See the [td_api.tl](https://github.com/tdlight-team/tdlight/blob/master/td/generate/scheme/td_api.tl) scheme or
the automatically generated [HTML documentation](https://core.telegram.org/tdlib/docs/td__api_8h.html) for a list of all available
TDLib [methods](https://core.telegram.org/tdlib/docs/classtd_1_1td__api_1_1_function.html) and [classes](https://core.telegram.org/tdlib/docs/classtd_1_1td__api_1_1_object.html).
The JSON representation of TDLib API objects is straightforward: all API objects are represented as JSON objects with the same keys as the API object field names in the
[td_api.tl](https://github.com/tdlib/td/blob/master/td/generate/scheme/td_api.tl) scheme. Note that in the automatically generated C++ documentation all fields have an additional terminating underscore
[td_api.tl](https://github.com/tdlight-team/tdlight/blob/master/td/generate/scheme/td_api.tl) scheme. Note that in the automatically generated C++ documentation all fields have an additional terminating underscore
which shouldn't be used in the JSON interface. The object type name is stored in the special field '@type' which is optional in places where type is uniquely determined by the context.
Fields of Bool type are stored as Boolean, fields of int32, int53, and double types are stored as Number, fields of int64 and string types are stored as String,
fields of bytes type are base64 encoded and then stored as String, fields of array type are stored as Array.

2
example/web/tdweb/src/index.js
View File

@ -65,7 +65,7 @@ class TdClient {
*
* If the query contains the field '@extra', the same field will be added into the result.
*
* @param {Object} query - The query for TDLib. See the [td_api.tl]{@link https://github.com/tdlib/td/blob/master/td/generate/scheme/td_api.tl} scheme or
* @param {Object} query - The query for TDLib. See the [td_api.tl]{@link https://github.com/tdlight-team/tdlight/blob/master/td/generate/scheme/td_api.tl} scheme or
* the automatically generated [HTML documentation]{@link https://core.telegram.org/tdlib/docs/td__api_8h.html}
* for a list of all available TDLib [methods]{@link https://core.telegram.org/tdlib/docs/classtd_1_1td__api_1_1_function.html} and
* [classes]{@link https://core.telegram.org/tdlib/docs/classtd_1_1td__api_1_1_object.html}.

BIN
info/memory-usage.jpg
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 173 KiB

24
manual_checks.md
View File

@ -1,24 +0,0 @@
# Manual checks
## After every merge
1. Find if any `get_channel_difference` has been added to `MessagesManager.cpp`, and replace it with `delayed_get_channel_difference`.
### Example
The following line
```cpp
get_channel_difference(dialog_id, old_pts, true, "add_pending_channel_update pts mismatch");
```
should become
```cpp
auto enable_reactive_channel_difference
= G()->shared_config()
.get_option_boolean("enable_reactive_channel_difference", false);
get_channel_difference_delayed(dialog_id, old_pts, true,
enable_reactive_channel_difference,
"add_pending_channel_update pts mismatch");
```
2. Find `make_unique<Dialog>` and check if there's `d.set_time()` after.

4
td/generate/scheme/td_api.tl
View File

@ -5408,10 +5408,6 @@ getDatabaseStatistics = DatabaseStatistics;
//@full Full memory statistics calculation
getMemoryStatistics full:Bool = MemoryStatistics;
//@description Optimize memory
//@full Full memory optimization
optimizeMemory full:Bool = Ok;
//@description Optimizes storage usage, i.e. deletes some files and returns new storage usage statistics. Secret thumbnails can't be deleted
//@size Limit on the total size of files after deletion, in bytes. Pass -1 to use the default limit
//@ttl Limit on the time that has passed since the last time a file was accessed (or creation time for some filesystems). Pass -1 to use the default limit

70
td/telegram/AnimationsManager.cpp
View File

@ -80,9 +80,7 @@ class SaveGifQuery final : public Td::ResultHandler {
}
void send(FileId file_id, tl_object_ptr<telegram_api::inputDocument> &&input_document, bool unsave) {
if (input_document == nullptr) {
return;
}
CHECK(input_document != nullptr);
CHECK(file_id.is_valid());
file_id_ = file_id;
file_reference_ = input_document->file_reference_.as_slice().str();
@ -146,17 +144,11 @@ AnimationsManager::AnimationsManager(Td *td, ActorShared<> parent) : td_(td), pa
void AnimationsManager::tear_down() {
parent_.reset();
if (td_->memory_manager_->can_manage_memory()) {
// Completely clear memory when closing, to avoid memory leaks
memory_cleanup(true);
}
}
int32 AnimationsManager::get_animation_duration(FileId file_id) const {
auto it = animations_.find(file_id);
if (it == animations_.end() || it->second == nullptr) {
return 0;
}
CHECK(it != animations_.end());
return it->second->duration;
}
@ -166,13 +158,9 @@ tl_object_ptr<td_api::animation> AnimationsManager::get_animation_object(FileId
}
auto it = animations_.find(file_id);
if (it == animations_.end()) {
return nullptr;
}
CHECK(it != animations_.end());
auto animation = it->second.get();
if (animation == nullptr) {
return nullptr;
}
CHECK(animation != nullptr);
auto thumbnail =
animation->animated_thumbnail.file_id.is_valid()
? get_thumbnail_object(td_->file_manager_.get(), animation->animated_thumbnail, PhotoFormat::Mpeg4)
@ -245,8 +233,8 @@ FileId AnimationsManager::on_get_animation(unique_ptr<Animation> new_animation,
const AnimationsManager::Animation *AnimationsManager::get_animation(FileId file_id) const {
auto animation = animations_.find(file_id);
if (animation == animations_.end() || animation->second == nullptr) {
return make_unique<Animation>().get();
if (animation == animations_.end()) {
return nullptr;
}
CHECK(animation->second->file_id == file_id);
@ -255,25 +243,19 @@ const AnimationsManager::Animation *AnimationsManager::get_animation(FileId file
FileId AnimationsManager::get_animation_thumbnail_file_id(FileId file_id) const {
auto animation = get_animation(file_id);
if (animation == nullptr) {
return FileId();
}
CHECK(animation != nullptr);
return animation->thumbnail.file_id;
}
FileId AnimationsManager::get_animation_animated_thumbnail_file_id(FileId file_id) const {
auto animation = get_animation(file_id);
if (animation == nullptr) {
return FileId();
}
CHECK(animation != nullptr);
return animation->animated_thumbnail.file_id;
}
void AnimationsManager::delete_animation_thumbnail(FileId file_id) {
auto &animation = animations_[file_id];
if (animation == nullptr) {
return;
}
CHECK(animation != nullptr);
animation->thumbnail = PhotoSize();
animation->animated_thumbnail = AnimationSize();
}
@ -302,7 +284,7 @@ void AnimationsManager::merge_animations(FileId new_id, FileId old_id, bool can_
bool need_merge = true;
auto new_it = animations_.find(new_id);
if (new_it == animations_.end() || new_it->second == nullptr) {
if (new_it == animations_.end()) {
auto &old = animations_[old_id];
if (!can_delete_old) {
dup_animation(new_id, old_id);
@ -368,9 +350,7 @@ tl_object_ptr<telegram_api::InputMedia> AnimationsManager::get_input_media(
if (input_file != nullptr) {
const Animation *animation = get_animation(file_id);
if (animation == nullptr) {
return nullptr;
}
CHECK(animation != nullptr);
vector<tl_object_ptr<telegram_api::DocumentAttribute>> attributes;
if (!animation->file_name.empty()) {
@ -411,9 +391,7 @@ SecretInputMedia AnimationsManager::get_secret_input_media(FileId animation_file
tl_object_ptr<telegram_api::InputEncryptedFile> input_file,
const string &caption, BufferSlice thumbnail) const {
auto *animation = get_animation(animation_file_id);
if (animation == nullptr) {
return SecretInputMedia{};
}
CHECK(animation != nullptr);
auto file_view = td_->file_manager_->get_file_view(animation_file_id);
auto &encryption_key = file_view.encryption_key();
if (!file_view.is_encrypted_secret() || encryption_key.empty()) {
@ -697,9 +675,7 @@ int64 AnimationsManager::get_saved_animations_hash(const char *source) const {
numbers.reserve(saved_animation_ids_.size());
for (auto animation_id : saved_animation_ids_) {
auto animation = get_animation(animation_id);
if (animation == nullptr) {
continue;
}
CHECK(animation != nullptr);
auto file_view = td_->file_manager_->get_file_view(animation_id);
CHECK(file_view.has_remote_location());
if (!file_view.remote_location().is_document()) {
@ -742,9 +718,6 @@ void AnimationsManager::send_save_gif_query(FileId animation_id, bool unsave, Pr
void AnimationsManager::add_saved_animation_by_id(FileId animation_id) {
auto animation = get_animation(animation_id);
if (animation == nullptr) {
return;
}
CHECK(animation != nullptr);
if (animation->has_stickers) {
return;
@ -883,7 +856,6 @@ void AnimationsManager::send_update_saved_animations(bool from_database) {
vector<FileId> new_saved_animation_file_ids = saved_animation_ids_;
for (auto &animation_id : saved_animation_ids_) {
auto animation = get_animation(animation_id);
if (animation != nullptr) {
CHECK(animation != nullptr);
if (animation->thumbnail.file_id.is_valid()) {
new_saved_animation_file_ids.push_back(animation->thumbnail.file_id);
@ -892,7 +864,6 @@ void AnimationsManager::send_update_saved_animations(bool from_database) {
new_saved_animation_file_ids.push_back(animation->animated_thumbnail.file_id);
}
}
}
std::sort(new_saved_animation_file_ids.begin(), new_saved_animation_file_ids.end());
if (new_saved_animation_file_ids != saved_animation_file_ids_) {
td_->file_manager_->change_files_source(get_saved_animations_file_source_id(), saved_animation_file_ids_,
@ -925,9 +896,7 @@ FileSourceId AnimationsManager::get_saved_animations_file_source_id() {
string AnimationsManager::get_animation_search_text(FileId file_id) const {
auto animation = get_animation(file_id);
if (animation == nullptr) {
return "";
}
CHECK(animation != nullptr);
return animation->file_name;
}
@ -951,17 +920,6 @@ void AnimationsManager::get_current_state(vector<td_api::object_ptr<td_api::Upda
}
}
void AnimationsManager::memory_cleanup() {
memory_cleanup(false);
}
void AnimationsManager::memory_cleanup(bool full) {
animations_.clear();
animations_.rehash(0);
saved_animation_ids_.clear();
saved_animation_file_ids_.clear();
}
void AnimationsManager::memory_stats(vector<string> &output) {
output.push_back("\"animations_\":"); output.push_back(std::to_string(animations_.size()));
output.push_back(",");

4
td/telegram/AnimationsManager.h
View File

@ -30,8 +30,6 @@ class AnimationsManager final : public Actor {
public:
AnimationsManager(Td *td, ActorShared<> parent);
void memory_cleanup();
void memory_stats(vector<string> &output);
int32 get_animation_duration(FileId file_id) const;
@ -115,8 +113,6 @@ class AnimationsManager final : public Actor {
FileId file_id;
};
void memory_cleanup(bool full);
const Animation *get_animation(FileId file_id) const;
FileId on_get_animation(unique_ptr<Animation> new_animation, bool replace);

4
td/telegram/AnimationsManager.hpp
View File

@ -22,9 +22,7 @@ namespace td {
template <class StorerT>
void AnimationsManager::store_animation(FileId file_id, StorerT &storer) const {
auto it = animations_.find(file_id);
if (it == animations_.end() || it->second == nullptr) {
return;
}
CHECK(it != animations_.end());
const Animation *animation = it->second.get();
bool has_animated_thumbnail = animation->animated_thumbnail.file_id.is_valid();
BEGIN_STORE_FLAGS();

43
td/telegram/AudiosManager.cpp
View File

@ -25,9 +25,7 @@ AudiosManager::AudiosManager(Td *td) : td_(td) {
int32 AudiosManager::get_audio_duration(FileId file_id) const {
auto it = audios_.find(file_id);
if (it == audios_.end() || it->second == nullptr) {
return 0;
}
CHECK(it != audios_.end());
return it->second->duration;
}
@ -37,13 +35,9 @@ tl_object_ptr<td_api::audio> AudiosManager::get_audio_object(FileId file_id) con
}
auto it = audios_.find(file_id);
if (it == audios_.end()) {
return nullptr;
}
CHECK(it != audios_.end());
auto audio = it->second.get();
if (audio == nullptr) {
return nullptr;
}
CHECK(audio != nullptr);
return make_tl_object<td_api::audio>(
audio->duration, audio->title, audio->performer, audio->file_name, audio->mime_type,
get_minithumbnail_object(audio->minithumbnail),
@ -93,8 +87,8 @@ FileId AudiosManager::on_get_audio(unique_ptr<Audio> new_audio, bool replace) {
const AudiosManager::Audio *AudiosManager::get_audio(FileId file_id) const {
auto audio = audios_.find(file_id);
if (audio == audios_.end() || audio->second == nullptr) {
return make_unique<Audio>().get();
if (audio == audios_.end()) {
return nullptr;
}
CHECK(audio->second->file_id == file_id);
@ -121,7 +115,7 @@ void AudiosManager::merge_audios(FileId new_id, FileId old_id, bool can_delete_o
CHECK(old_ != nullptr);
auto new_it = audios_.find(new_id);
if (new_it == audios_.end() || new_it->second == nullptr) {
if (new_it == audios_.end()) {
auto &old = audios_[old_id];
if (!can_delete_old) {
dup_audio(new_id, old_id);
@ -149,25 +143,19 @@ void AudiosManager::merge_audios(FileId new_id, FileId old_id, bool can_delete_o
string AudiosManager::get_audio_search_text(FileId file_id) const {
auto audio = get_audio(file_id);
if (audio == nullptr) {
return "";
}
CHECK(audio != nullptr);
return PSTRING() << audio->file_name << " " << audio->title << " " << audio->performer;
}
FileId AudiosManager::get_audio_thumbnail_file_id(FileId file_id) const {
auto audio = get_audio(file_id);
if (audio == nullptr) {
return FileId();
}
CHECK(audio != nullptr);
return audio->thumbnail.file_id;
}
void AudiosManager::delete_audio_thumbnail(FileId file_id) {
auto &audio = audios_[file_id];
if (audio == nullptr) {
return;
}
CHECK(audio != nullptr);
audio->thumbnail = PhotoSize();
}
@ -191,9 +179,7 @@ SecretInputMedia AudiosManager::get_secret_input_media(FileId audio_file_id,
tl_object_ptr<telegram_api::InputEncryptedFile> input_file,
const string &caption, BufferSlice thumbnail) const {
auto *audio = get_audio(audio_file_id);
if (audio == nullptr) {
return SecretInputMedia{};
}
CHECK(audio != nullptr);
auto file_view = td_->file_manager_->get_file_view(audio_file_id);
auto &encryption_key = file_view.encryption_key();
if (!file_view.is_encrypted_secret() || encryption_key.empty()) {
@ -241,9 +227,7 @@ tl_object_ptr<telegram_api::InputMedia> AudiosManager::get_input_media(
if (input_file != nullptr) {
const Audio *audio = get_audio(file_id);
if (audio == nullptr) {
return nullptr;
}
CHECK(audio != nullptr);
vector<tl_object_ptr<telegram_api::DocumentAttribute>> attributes;
attributes.push_back(make_tl_object<telegram_api::documentAttributeAudio>(
@ -269,10 +253,7 @@ tl_object_ptr<telegram_api::InputMedia> AudiosManager::get_input_media(
return nullptr;
}
void AudiosManager::memory_cleanup() {
audios_.clear();
audios_.rehash(0);
}
void AudiosManager::memory_stats(vector<string> &output) {
output.push_back("\"audios_\":"); output.push_back(std::to_string(audios_.size()));

2
td/telegram/AudiosManager.h
View File

@ -26,8 +26,6 @@ class AudiosManager {
public:
explicit AudiosManager(Td *td);
void memory_cleanup();
void memory_stats(vector<string> &output);
int32 get_audio_duration(FileId file_id) const;

4
td/telegram/AudiosManager.hpp
View File

@ -22,9 +22,7 @@ namespace td {
template <class StorerT>
void AudiosManager::store_audio(FileId file_id, StorerT &storer) const {
auto it = audios_.find(file_id);
if (it == audios_.end() || it->second == nullptr) {
return;
}
CHECK(it != audios_.end());
const Audio *audio = it->second.get();
store(audio->file_name, storer);
store(audio->mime_type, storer);

44
td/telegram/BackgroundManager.cpp
View File

@ -140,9 +140,7 @@ class UploadBackgroundQuery final : public Td::ResultHandler {
void send(FileId file_id, tl_object_ptr<telegram_api::InputFile> &&input_file, const BackgroundType &type,
bool for_dark_theme) {
if (input_file == nullptr) {
return;
}
CHECK(input_file != nullptr);
file_id_ = file_id;
type_ = type;
for_dark_theme_ = for_dark_theme;
@ -773,7 +771,7 @@ void BackgroundManager::save_background_id(bool for_dark_theme) {
auto background_id = set_background_id_[for_dark_theme];
if (background_id.is_valid()) {
const Background *background = get_background(background_id);
if (!(background != nullptr)) return;
CHECK(background != nullptr);
BackgroundLogEvent log_event{*background, set_background_type_[for_dark_theme]};
G()->td_db()->get_binlog_pmc()->set(key, log_event_store(log_event).as_slice().str());
} else {
@ -826,9 +824,7 @@ void BackgroundManager::on_upload_background_file(FileId file_id, tl_object_ptr<
LOG(INFO) << "Background file " << file_id << " has been uploaded";
auto it = being_uploaded_files_.find(file_id);
if (it == being_uploaded_files_.end()) {
return;
}
CHECK(it != being_uploaded_files_.end());
auto type = it->second.type;
auto for_dark_theme = it->second.for_dark_theme;
@ -849,9 +845,7 @@ void BackgroundManager::on_upload_background_file_error(FileId file_id, Status s
CHECK(status.is_error());
auto it = being_uploaded_files_.find(file_id);
if (it == being_uploaded_files_.end()) {
return;
}
CHECK(it != being_uploaded_files_.end());
auto promise = std::move(it->second.promise);
@ -882,7 +876,7 @@ void BackgroundManager::do_upload_background_file(FileId file_id, const Backgrou
void BackgroundManager::on_uploaded_background_file(FileId file_id, const BackgroundType &type, bool for_dark_theme,
telegram_api::object_ptr<telegram_api::WallPaper> wallpaper,
Promise<Unit> &&promise) {
if (!(wallpaper != nullptr)) promise.set_error(Status::Error(500, "Error"));
CHECK(wallpaper != nullptr);
auto added_background = on_get_background(BackgroundId(), string(), std::move(wallpaper), true);
auto background_id = added_background.first;
@ -894,7 +888,7 @@ void BackgroundManager::on_uploaded_background_file(FileId file_id, const Backgr
<< "Type of uploaded background has changed from " << type << " to " << added_background.second;
const auto *background = get_background(background_id);
if (!(background != nullptr)) return promise.set_error(Status::Error(500, "Error"));
CHECK(background != nullptr);
if (!background->file_id.is_valid()) {
td_->file_manager_->cancel_upload(file_id);
return promise.set_error(Status::Error(500, "Receive wrong uploaded background without file"));
@ -966,12 +960,6 @@ void BackgroundManager::on_reset_background(Result<Unit> &&result, Promise<Unit>
if (result.is_error()) {
return promise.set_error(result.move_as_error());
}
reset_backgrounds_data();
promise.set_value(Unit());
}
void BackgroundManager::reset_backgrounds_data() {
installed_backgrounds_.clear();
set_background_id(BackgroundId(), BackgroundType(), false);
set_background_id(BackgroundId(), BackgroundType(), true);
@ -983,6 +971,8 @@ void BackgroundManager::reset_backgrounds_data() {
local_background_ids_[1].clear();
save_local_backgrounds(true);
}
promise.set_value(Unit());
}
void BackgroundManager::add_background(const Background &background, bool replace_type) {
@ -1304,24 +1294,6 @@ void BackgroundManager::get_current_state(vector<td_api::object_ptr<td_api::Upda
updates.push_back(get_update_selected_background_object(true));
}
void BackgroundManager::memory_cleanup() {
memory_cleanup(false);
}
void BackgroundManager::memory_cleanup(bool full) {
backgrounds_.clear();
backgrounds_.rehash(0);
background_id_to_file_source_id_.clear();
background_id_to_file_source_id_.rehash(0);
name_to_background_id_.clear();
name_to_background_id_.rehash(0);
file_id_to_background_id_.clear();
file_id_to_background_id_.rehash(0);
loaded_from_database_backgrounds_.clear();
loaded_from_database_backgrounds_.rehash(0);
reset_backgrounds_data();
}
void BackgroundManager::memory_stats(vector<string> &output) {
output.push_back("\"backgrounds_\":"); output.push_back(std::to_string(backgrounds_.size()));
output.push_back(",");

6
td/telegram/BackgroundManager.h
View File

@ -34,8 +34,6 @@ class BackgroundManager final : public Actor {
public:
BackgroundManager(Td *td, ActorShared<> parent);
void memory_cleanup();
void memory_stats(vector<string> &output);
void get_backgrounds(bool for_dark_theme, Promise<td_api::object_ptr<td_api::backgrounds>> &&promise);
@ -105,8 +103,6 @@ class BackgroundManager final : public Actor {
void tear_down() final;
void memory_cleanup(bool full);
static string get_background_database_key(bool for_dark_theme);
static string get_local_backgrounds_database_key(bool for_dark_theme);
@ -157,8 +153,6 @@ class BackgroundManager final : public Actor {
void on_reset_background(Result<Unit> &&result, Promise<Unit> &&promise);
void reset_backgrounds_data();
void upload_background_file(FileId file_id, const BackgroundType &type, bool for_dark_theme, Promise<Unit> &&promise);
void on_upload_background_file(FileId file_id, tl_object_ptr<telegram_api::InputFile> input_file);

4
td/telegram/CallbackQueriesManager.cpp
View File

@ -44,9 +44,7 @@ class GetBotCallbackAnswerQuery final : public Td::ResultHandler {
message_id_ = message_id;
auto input_peer = td->messages_manager_->get_input_peer(dialog_id, AccessRights::Read);
if (input_peer == nullptr) {
return;
}
CHECK(input_peer != nullptr);
int32 flags = 0;
BufferSlice data;

14
td/telegram/ChannelId.h
View File

@ -11,7 +11,6 @@
#include "td/utils/common.h"
#include "td/utils/StringBuilder.h"
#include <ctime>
#include <functional>
#include <type_traits>
@ -19,7 +18,6 @@ namespace td {
class ChannelId {
int64 id = 0;
int64 time_ = INT64_MAX;
public:
// the last (1 << 31) - 1 identifiers will be used for secret chat dialog identifiers
@ -40,18 +38,6 @@ class ChannelId {
return id;
}
void set_time() {
time_ = std::time(nullptr);
}
int64 get_time() const {
return time_;
}
void reset_time() {
time_ = INT64_MAX;
}
bool operator==(const ChannelId &other) const {
return id == other.id;
}

19
td/telegram/ChatId.h
View File

@ -11,7 +11,6 @@
#include "td/utils/common.h"
#include "td/utils/StringBuilder.h"
#include <ctime>
#include <functional>
#include <type_traits>
@ -19,17 +18,13 @@ namespace td {
class ChatId {
int64 id = 0;
int64 time_ = INT64_MAX;
public:
static constexpr int64 MAX_CHAT_ID = 999999999999ll;
explicit ChatId() {
set_time();
};
ChatId() = default;
explicit ChatId(int64 chat_id) : id(chat_id) {
set_time();
}
template <class T, typename = std::enable_if_t<std::is_convertible<T, int64>::value>>
ChatId(T chat_id) = delete;
@ -42,18 +37,6 @@ class ChatId {
return id;
}
void set_time() {
time_ = std::time(nullptr);
}
int64 get_time() const {
return time_;
}
void reset_time() {
time_ = INT64_MAX;
}
bool operator==(const ChatId &other) const {
return id == other.id;
}

1
td/telegram/Client.cpp
View File

@ -8,7 +8,6 @@
#include "td/telegram/Td.h"
#include "td/telegram/TdCallback.h"
#include "td/telegram/Log.h"
#include "td/actor/actor.h"
#include "td/actor/ConcurrentScheduler.h"

218
td/telegram/ContactsManager.cpp
View File

@ -3427,10 +3427,6 @@ ContactsManager::~ContactsManager() = default;
void ContactsManager::tear_down() {
parent_.reset();
if (td_->memory_manager_->can_manage_memory()) {
// Completely clear memory when closing, to avoid memory leaks
memory_cleanup(true);
}
}
UserId ContactsManager::load_my_id() {
@ -3468,7 +3464,7 @@ void ContactsManager::on_user_online_timeout(UserId user_id) {
}
auto u = get_user(user_id);
if (u == nullptr) { return; }
CHECK(u != nullptr);
CHECK(u->is_update_user_sent);
LOG(INFO) << "Update " << user_id << " online status to offline";
@ -3494,7 +3490,7 @@ void ContactsManager::on_channel_unban_timeout(ChannelId channel_id) {
}
auto c = get_channel(channel_id);
if (c == nullptr) { return; }
CHECK(c != nullptr);
auto old_status = c->status;
c->status.update_restrictions();
@ -3527,7 +3523,7 @@ void ContactsManager::on_user_nearby_timeout(UserId user_id) {
}
auto u = get_user(user_id);
if (u == nullptr) { return; }
CHECK(u != nullptr);
LOG(INFO) << "Remove " << user_id << " from nearby list";
DialogId dialog_id(user_id);
@ -10097,14 +10093,10 @@ void ContactsManager::update_chat_full(ChatFull *chat_full, ChatId chat_id, cons
void ContactsManager::update_channel_full(ChannelFull *channel_full, ChannelId channel_id, const char *source,
bool from_database) {
if (channel_full == nullptr) {
return;
}
CHECK(channel_full != nullptr);
unavailable_channel_fulls_.erase(channel_id); // don't needed anymore
if (!(channel_full->participant_count >= channel_full->administrator_count)) {
return;
}
CHECK(channel_full->participant_count >= channel_full->administrator_count);
if (channel_full->is_slow_mode_next_send_date_changed) {
auto now = G()->server_time();
@ -10143,9 +10135,7 @@ void ContactsManager::update_channel_full(ChannelFull *channel_full, ChannelId c
{
Channel *c = get_channel(channel_id);
if (!(c == nullptr || c->is_update_supergroup_sent)) {
return;
}
CHECK(c == nullptr || c->is_update_supergroup_sent);
}
if (!channel_full->is_update_channel_full_sent) {
LOG(ERROR) << "Send partial updateSupergroupFullInfo for " << channel_id << " from " << source;
@ -11475,9 +11465,8 @@ void ContactsManager::update_user_online_member_count(User *u) {
case DialogType::Chat: {
auto chat_id = dialog_id.get_chat_id();
auto chat_full = get_chat_full(chat_id);
if (chat_full != nullptr) {
CHECK(chat_full != nullptr);
update_chat_online_member_count(chat_full, chat_id, false);
}
break;
}
case DialogType::Channel: {
@ -13661,8 +13650,6 @@ Result<ContactsManager::BotData> ContactsManager::get_bot_data(UserId user_id) c
return Status::Error(400, "Bot is inaccessible");
}
user_id.set_time(); // update last access time of UserId
BotData bot_data;
bot_data.username = bot->username;
bot_data.can_join_groups = bot->can_join_groups;
@ -13692,7 +13679,6 @@ const ContactsManager::User *ContactsManager::get_user(UserId user_id) const {
if (p == users_.end()) {
return nullptr;
} else {
user_id.set_time(); // update last access time of UserId
return p->second.get();
}
}
@ -13702,7 +13688,6 @@ ContactsManager::User *ContactsManager::get_user(UserId user_id) {
if (p == users_.end()) {
return nullptr;
} else {
user_id.set_time(); // update last access time of UserId
return p->second.get();
}
}
@ -13787,7 +13772,6 @@ bool ContactsManager::get_user(UserId user_id, int left_tries, Promise<Unit> &&p
return false;
}
user_id.set_time();
promise.set_value(Unit());
return true;
}
@ -13798,7 +13782,6 @@ ContactsManager::User *ContactsManager::add_user(UserId user_id, const char *sou
if (user_ptr == nullptr) {
user_ptr = make_unique<User>();
}
user_id.set_time();
return user_ptr.get();
}
@ -13807,7 +13790,6 @@ const ContactsManager::UserFull *ContactsManager::get_user_full(UserId user_id)
if (p == users_full_.end()) {
return nullptr;
} else {
user_id.set_time();
return p->second.get();
}
}
@ -13817,7 +13799,6 @@ ContactsManager::UserFull *ContactsManager::get_user_full(UserId user_id) {
if (p == users_full_.end()) {
return nullptr;
} else {
user_id.set_time();
return p->second.get();
}
}
@ -13828,7 +13809,6 @@ ContactsManager::UserFull *ContactsManager::add_user_full(UserId user_id) {
if (user_full_ptr == nullptr) {
user_full_ptr = make_unique<UserFull>();
}
user_id.set_time();
return user_full_ptr.get();
}
@ -13842,7 +13822,6 @@ void ContactsManager::reload_user(UserId user_id, Promise<Unit> &&promise) {
if (input_user == nullptr) {
return promise.set_error(Status::Error(400, "User info not found"));
}
user_id.set_time();
// there is no much reason to combine different requests into one request
vector<tl_object_ptr<telegram_api::InputUser>> users;
@ -13875,14 +13854,12 @@ void ContactsManager::load_user_full(UserId user_id, bool force, Promise<Unit> &
send_get_user_full_query(user_id, std::move(input_user), Auto(), "load expired user_full");
}
user_id.set_time();
promise.set_value(Unit());
}
void ContactsManager::reload_user_full(UserId user_id) {
auto input_user = get_input_user(user_id);
if (input_user != nullptr) {
user_id.set_time();
send_get_user_full_query(user_id, std::move(input_user), Auto(), "reload_user_full");
}
}
@ -14039,7 +14016,6 @@ const ContactsManager::Chat *ContactsManager::get_chat(ChatId chat_id) const {
if (p == chats_.end()) {
return nullptr;
} else {
chat_id.set_time(); // update last access time of ChatId
return p->second.get();
}
}
@ -14049,7 +14025,6 @@ ContactsManager::Chat *ContactsManager::get_chat(ChatId chat_id) {
if (p == chats_.end()) {
return nullptr;
} else {
chat_id.set_time(); // update last access time of ChatId
return p->second.get();
}
}
@ -14376,7 +14351,6 @@ const ContactsManager::Channel *ContactsManager::get_channel(ChannelId channel_i
if (p == channels_.end()) {
return nullptr;
} else {
channel_id.set_time(); // update last access time of ChannelId
return p->second.get();
}
}
@ -14386,7 +14360,6 @@ ContactsManager::Channel *ContactsManager::get_channel(ChannelId channel_id) {
if (p == channels_.end()) {
return nullptr;
} else {
channel_id.set_time(); // update last access time of ChannelId
return p->second.get();
}
}
@ -15904,7 +15877,7 @@ td_api::object_ptr<td_api::updateUser> ContactsManager::get_update_unknown_user_
int64 ContactsManager::get_user_id_object(UserId user_id, const char *source) const {
if (user_id.is_valid() && get_user(user_id) == nullptr && unknown_users_.count(user_id) == 0) {
LOG(WARNING) << "Have no info about " << user_id << " from " << source;
LOG(ERROR) << "Have no info about " << user_id << " from " << source;
unknown_users_.insert(user_id);
send_closure(G()->td(), &Td::send_update, get_update_unknown_user_object(user_id));
}
@ -15998,7 +15971,7 @@ td_api::object_ptr<td_api::updateBasicGroup> ContactsManager::get_update_unknown
int64 ContactsManager::get_basic_group_id_object(ChatId chat_id, const char *source) const {
if (chat_id.is_valid() && get_chat(chat_id) == nullptr && unknown_chats_.count(chat_id) == 0) {
LOG(WARNING) << "Have no info about " << chat_id << " from " << source;
LOG(ERROR) << "Have no info about " << chat_id << " from " << source;
unknown_chats_.insert(chat_id);
send_closure(G()->td(), &Td::send_update, get_update_unknown_basic_group_object(chat_id));
}
@ -16052,7 +16025,7 @@ td_api::object_ptr<td_api::updateSupergroup> ContactsManager::get_update_unknown
int64 ContactsManager::get_supergroup_id_object(ChannelId channel_id, const char *source) const {
if (channel_id.is_valid() && get_channel(channel_id) == nullptr && unknown_channels_.count(channel_id) == 0) {
LOG(WARNING) << "Have no info about " << channel_id << " received from " << source;
LOG(ERROR) << "Have no info about " << channel_id << " received from " << source;
unknown_channels_.insert(channel_id);
send_closure(G()->td(), &Td::send_update, get_update_unknown_supergroup_object(channel_id));
}
@ -16080,9 +16053,7 @@ tl_object_ptr<td_api::supergroupFullInfo> ContactsManager::get_supergroup_full_i
tl_object_ptr<td_api::supergroupFullInfo> ContactsManager::get_supergroup_full_info_object(
const ChannelFull *channel_full, ChannelId channel_id) const {
if(channel_full == nullptr) {
return nullptr;
}
CHECK(channel_full != nullptr);
double slow_mode_delay_expires_in = 0;
if (channel_full->slow_mode_next_send_date != 0) {
slow_mode_delay_expires_in = max(channel_full->slow_mode_next_send_date - G()->server_time(), 1e-3);
@ -16127,7 +16098,7 @@ td_api::object_ptr<td_api::updateSecretChat> ContactsManager::get_update_unknown
int32 ContactsManager::get_secret_chat_id_object(SecretChatId secret_chat_id, const char *source) const {
if (secret_chat_id.is_valid() && get_secret_chat(secret_chat_id) == nullptr &&
unknown_secret_chats_.count(secret_chat_id) == 0) {
LOG(WARNING) << "Have no info about " << secret_chat_id << " from " << source;
LOG(ERROR) << "Have no info about " << secret_chat_id << " from " << source;
unknown_secret_chats_.insert(secret_chat_id);
send_closure(G()->td(), &Td::send_update, get_update_unknown_secret_chat_object(secret_chat_id));
}
@ -16314,171 +16285,6 @@ void ContactsManager::get_current_state(vector<td_api::object_ptr<td_api::Update
}
}
void ContactsManager::memory_cleanup() {
memory_cleanup(false);
}
void ContactsManager::memory_cleanup(bool full) {
auto time = std::time(nullptr);
auto user_ttl = full ? 0 : !G()->shared_config().get_option_integer("delete_user_reference_after_seconds", 3600);
auto chat_ttl = full ? 0 : !G()->shared_config().get_option_integer("delete_chat_reference_after_seconds", 3600);
auto chat_access_hash_cleanup = full ? true : !G()->shared_config().get_option_boolean("experiment_enable_chat_access_hash_cleanup", true);
/* DESTROY INVALID USERS */
if (full) {
users_.clear();
} else {
auto it = users_.begin();
while (it != users_.end()) {
//auto &user = it->second;
auto user_id = it->first;
auto is_invalid = time - user_id.get_time() > user_ttl;
if (is_invalid) {
user_id.reset_time();
it = users_.erase(it);
} else {
it++;
}
}
}
users_.rehash(0);
users_full_.clear();
users_full_.rehash(0);
user_photos_.clear();
user_photos_.rehash(0);
unknown_users_.clear();
unknown_users_.rehash(0);
user_profile_photo_file_source_ids_.clear();
user_profile_photo_file_source_ids_.rehash(0);
my_photo_file_id_.clear();
my_photo_file_id_.rehash(0);
if (full || chat_access_hash_cleanup) {
/* DESTROY INVALID CHATS */
if (full) {
chats_.clear();
} else {
auto it = chats_.begin();
while (it != chats_.end()) {
//auto &chat = it->second;
auto chat_id = it->first;
auto is_invalid = time - chat_id.get_time() > chat_ttl;
if (is_invalid) {
chat_id.reset_time();
it = chats_.erase(it);
} else {
it++;
}
}
}
chats_.rehash(0);
}
chats_full_.clear();
chats_full_.rehash(0);
unknown_chats_.clear();
unknown_chats_.rehash(0);
if (full || chat_access_hash_cleanup) {
chat_full_file_source_ids_.clear();
chat_full_file_source_ids_.rehash(0);
min_channels_.clear();
min_channels_.rehash(0);
/* DESTROY INVALID CHANNELS */
if (full) {
channels_.clear();
} else {
auto it = channels_.begin();
while (it != channels_.end()) {
//auto &channel = it->second;
auto channel_id = it->first;
auto is_invalid = time - channel_id.get_time() > chat_ttl;
if (is_invalid) {
channel_id.reset_time();
it = channels_.erase(it);
} else {
it++;
}
}
}
channels_.rehash(0);
channels_full_.clear();
channels_full_.rehash(0);
}
unknown_channels_.clear();
unknown_channels_.rehash(0);
if (full || chat_access_hash_cleanup) {
channel_full_file_source_ids_.clear();
channel_full_file_source_ids_.rehash(0);
}
if (full) {
secret_chats_.clear();
secret_chats_.rehash(0);
unknown_secret_chats_.clear();
unknown_secret_chats_.rehash(0);
secret_chats_with_user_.clear();
secret_chats_with_user_.rehash(0);
}
invite_link_infos_.clear();
invite_link_infos_.rehash(0);
dialog_access_by_invite_link_.clear();
dialog_access_by_invite_link_.rehash(0);
load_user_from_database_queries_.clear();
load_user_from_database_queries_.rehash(0);
loaded_from_database_users_.clear();
loaded_from_database_users_.rehash(0);
unavailable_user_fulls_.clear();
unavailable_user_fulls_.rehash(0);
load_chat_from_database_queries_.clear();
load_chat_from_database_queries_.rehash(0);
loaded_from_database_chats_.clear();
loaded_from_database_chats_.rehash(0);
unavailable_chat_fulls_.clear();
unavailable_chat_fulls_.rehash(0);
load_channel_from_database_queries_.clear();
load_channel_from_database_queries_.rehash(0);
loaded_from_database_channels_.clear();
loaded_from_database_channels_.rehash(0);
unavailable_channel_fulls_.clear();
unavailable_channel_fulls_.rehash(0);
if (full) {
load_secret_chat_from_database_queries_.clear();
load_secret_chat_from_database_queries_.rehash(0);
loaded_from_database_secret_chats_.clear();
loaded_from_database_secret_chats_.rehash(0);
}
dialog_administrators_.clear();
dialog_administrators_.rehash(0);
uploaded_profile_photos_.clear();
uploaded_profile_photos_.rehash(0);
imported_contacts_.clear();
imported_contacts_.rehash(0);
cached_channel_participants_.clear();
cached_channel_participants_.rehash(0);
load_contacts_queries_.clear();
load_imported_contacts_queries_.clear();
all_imported_contacts_.clear();
users_nearby_.clear();
channels_nearby_.clear();
all_users_nearby_.clear();
all_users_nearby_.rehash(0);
linked_channel_ids_.clear();
linked_channel_ids_.rehash(0);
restricted_user_ids_.clear();
restricted_user_ids_.rehash(0);
restricted_channel_ids_.clear();
restricted_channel_ids_.rehash(0);
next_all_imported_contacts_.clear();
imported_contact_user_ids_.clear();
unimported_contact_invites_.clear();
}
void ContactsManager::memory_stats(vector<string> &output) {
output.push_back("\"users_\":"); output.push_back(std::to_string(users_.size()));

4
td/telegram/ContactsManager.h
View File

@ -68,8 +68,6 @@ class ContactsManager final : public Actor {
static UserId load_my_id();
void memory_cleanup();
void memory_stats(vector<string> &output);
static UserId get_user_id(const tl_object_ptr<telegram_api::User> &user);
@ -1094,8 +1092,6 @@ class ContactsManager final : public Actor {
static constexpr int32 ACCOUNT_UPDATE_LAST_NAME = 1 << 1;
static constexpr int32 ACCOUNT_UPDATE_ABOUT = 1 << 2;
void memory_cleanup(bool full);
static bool have_input_peer_user(const User *u, AccessRights access_rights);
static bool have_input_peer_chat(const Chat *c, AccessRights access_rights);
bool have_input_peer_channel(const Channel *c, ChannelId channel_id, AccessRights access_rights,

3
td/telegram/Document.cpp
View File

@ -13,9 +13,10 @@
#include "td/telegram/Td.h"
#include "td/telegram/VideoNotesManager.h"
#include "td/telegram/VideosManager.h"
#include "td/telegram/Global.h"
#include "td/telegram/ConfigShared.h"
#include "td/utils/algorithm.h"
#include "Document.hpp"
namespace td {

8
td/telegram/Document.hpp
View File

@ -33,9 +33,7 @@ namespace td {
template <class StorerT>
void store(const Document &document, StorerT &storer) {
Td *td = storer.context()->td().get_actor_unsafe();
if (td == nullptr) {
return;
}
CHECK(td != nullptr);
store(document.type, storer);
switch (document.type) {
@ -69,9 +67,7 @@ void store(const Document &document, StorerT &storer) {
template <class ParserT>
void parse(Document &document, ParserT &parser) {
Td *td = parser.context()->td().get_actor_unsafe();
if (td == nullptr) {
return;
}
CHECK(td != nullptr);
parse(document.type, parser);
switch (document.type) {

50
td/telegram/DocumentsManager.cpp
View File

@ -52,11 +52,10 @@ tl_object_ptr<td_api::document> DocumentsManager::get_document_object(FileId fil
return nullptr;
}
auto document_it = documents_.find(file_id);
if (document_it == documents_.end() || document_it->second == nullptr) {
return nullptr;
}
auto document = document_it->second.get();
auto it = documents_.find(file_id);
CHECK(it != documents_.end());
auto document = it->second.get();
CHECK(document != nullptr);
return make_tl_object<td_api::document>(
document->file_name, document->mime_type, get_minithumbnail_object(document->minithumbnail),
get_thumbnail_object(td_->file_manager_.get(), document->thumbnail, thumbnail_format),
@ -114,7 +113,7 @@ Document DocumentsManager::on_get_document(RemoteDocument remote_document, Dialo
auto video_dimensions = get_dimensions(video->w_, video->h_, "documentAttributeVideo");
if (dimensions.width == 0 || (video_dimensions.width != 0 && video_dimensions != dimensions)) {
if (dimensions.width != 0) {
LOG(INFO) << "Receive ambiguous video dimensions " << dimensions << " and " << video_dimensions;
LOG(ERROR) << "Receive ambiguous video dimensions " << dimensions << " and " << video_dimensions;
}
dimensions = video_dimensions;
}
@ -524,13 +523,11 @@ void DocumentsManager::create_document(FileId file_id, string minithumbnail, Pho
const DocumentsManager::GeneralDocument *DocumentsManager::get_document(FileId file_id) const {
auto document = documents_.find(file_id);
if (document == documents_.end() ||
document->second == nullptr ||
document->second->file_id != file_id) {
return make_unique<GeneralDocument>().get();
if (document == documents_.end()) {
return nullptr;
}
CHECK(document->second->file_id == file_id);
return document->second.get();
}
@ -560,9 +557,7 @@ SecretInputMedia DocumentsManager::get_secret_input_media(FileId document_file_i
tl_object_ptr<telegram_api::InputEncryptedFile> input_file,
const string &caption, BufferSlice thumbnail) const {
const GeneralDocument *document = get_document(document_file_id);
if (document == nullptr) {
return SecretInputMedia{};
}
CHECK(document != nullptr);
auto file_view = td_->file_manager_->get_file_view(document_file_id);
auto &encryption_key = file_view.encryption_key();
if (!file_view.is_encrypted_secret() || encryption_key.empty()) {
@ -606,9 +601,7 @@ tl_object_ptr<telegram_api::InputMedia> DocumentsManager::get_input_media(
if (input_file != nullptr) {
const GeneralDocument *document = get_document(file_id);
if (document == nullptr) {
return nullptr;
}
CHECK(document != nullptr);
vector<tl_object_ptr<telegram_api::DocumentAttribute>> attributes;
if (document->file_name.size()) {
@ -633,18 +626,14 @@ tl_object_ptr<telegram_api::InputMedia> DocumentsManager::get_input_media(
FileId DocumentsManager::get_document_thumbnail_file_id(FileId file_id) const {
auto document = get_document(file_id);
if (document == nullptr) {
return FileId();
}
CHECK(document != nullptr);
return document->thumbnail.file_id;
}
void DocumentsManager::delete_document_thumbnail(FileId file_id) {
auto document_it = documents_.find(file_id);
if (document_it == documents_.end() || document_it->second == nullptr) {
return;
}
document_it->second->thumbnail = PhotoSize();
auto &document = documents_[file_id];
CHECK(document != nullptr);
document->thumbnail = PhotoSize();
}
FileId DocumentsManager::dup_document(FileId new_id, FileId old_id) {
@ -667,17 +656,14 @@ void DocumentsManager::merge_documents(FileId new_id, FileId old_id, bool can_de
CHECK(old_ != nullptr);
auto new_it = documents_.find(new_id);
if (new_it == documents_.end() || new_it->second == nullptr) {
auto old_it = documents_.find(old_id);
if (old_it != documents_.end() && old_it->second != nullptr) {
auto &old = old_it->second;
if (new_it == documents_.end()) {
auto &old = documents_[old_id];
if (!can_delete_old) {
dup_document(new_id, old_id);
} else {
old->file_id = new_id;
documents_.emplace(new_id, std::move(old));
}
}
} else {
GeneralDocument *new_ = new_it->second.get();
CHECK(new_ != nullptr);
@ -692,10 +678,6 @@ void DocumentsManager::merge_documents(FileId new_id, FileId old_id, bool can_de
}
}
void DocumentsManager::memory_cleanup() {
documents_.clear();
documents_.rehash(0);
}
void DocumentsManager::memory_stats(vector<string> &output) {
output.push_back("\"documents_\":"); output.push_back(std::to_string(documents_.size()));
}

2
td/telegram/DocumentsManager.h
View File

@ -77,8 +77,6 @@ class DocumentsManager {
tl_object_ptr<td_api::document> get_document_object(FileId file_id, PhotoFormat thumbnail_format) const;
void memory_cleanup();
void memory_stats(vector<string> &output);
Document on_get_document(RemoteDocument remote_document, DialogId owner_dialog_id,

10
td/telegram/DocumentsManager.hpp
View File

@ -21,15 +21,10 @@ namespace td {
template <class StorerT>
void DocumentsManager::store_document(FileId file_id, StorerT &storer) const {
//LOG(DEBUG) << "Store document " << file_id;
LOG(DEBUG) << "Store document " << file_id;
auto it = documents_.find(file_id);
if (it == documents_.end() || it->second == nullptr) {
return;
}
CHECK(it != documents_.end());
const GeneralDocument *document = it->second.get();
if (document == nullptr) {
return;
}
store(document->file_name, storer);
store(document->mime_type, storer);
store(document->minithumbnail, storer);
@ -65,7 +60,6 @@ FileId DocumentsManager::parse_document(ParserT &parser) {
parse(document->thumbnail, parser);
parse(document->file_id, parser);
LOG(DEBUG) << "Parsed document " << document->file_id;
if (parser.get_error() != nullptr || !document->file_id.is_valid()) {
return FileId();

91
td/telegram/FileReferenceManager.cpp
View File

@ -61,15 +61,14 @@ fileSourceSupergroupFull supergroup_id:int32 = FileSource; // repa
*/
FileSourceId FileReferenceManager::get_current_file_source_id() const {
return FileSourceId(narrow_cast<int32>((int32) unique_file_source_id));
return FileSourceId(narrow_cast<int32>(file_sources_.size()));
}
template <class T>
FileSourceId FileReferenceManager::add_file_source_id(T source, Slice source_str) {
auto source_id = ++unique_file_source_id;
file_sources_[source_id] = (std::move(source));
VLOG(file_references) << "Create file source " << source_id << " for " << source_str;
return FileSourceId(narrow_cast<int32>((int32) source_id));
file_sources_.emplace_back(std::move(source));
VLOG(file_references) << "Create file source " << file_sources_.size() << " for " << source_str;
return get_current_file_source_id();
}
FileSourceId FileReferenceManager::create_message_file_source(FullMessageId full_message_id) {
@ -142,20 +141,14 @@ vector<FileSourceId> FileReferenceManager::get_some_file_sources(NodeId node_id)
return it->second.file_source_ids.get_some_elements();
}
vector<FileSourceId> FileReferenceManager::get_all_file_sources(NodeId node_id) {
auto it = nodes_.find(node_id);
if (it == nodes_.end()) {
return {};
}
return it->second.file_source_ids.get_all_elements();
}
vector<FullMessageId> FileReferenceManager::get_some_message_file_sources(NodeId node_id) {
auto file_source_ids = get_some_file_sources(node_id);
vector<FullMessageId> result;
for (auto file_source_id : file_source_ids) {
const auto &file_source = file_sources_[file_source_id.get()];
auto index = static_cast<size_t>(file_source_id.get()) - 1;
CHECK(index < file_sources_.size());
const auto &file_source = file_sources_[index];
if (file_source.get_offset() == 0) {
result.push_back(file_source.get<FileSourceMessage>().full_message_id);
}
@ -258,7 +251,9 @@ void FileReferenceManager::send_query(Destination dest, FileSourceId file_source
send_closure(file_manager, &FileManager::on_file_reference_repaired, dest.node_id, file_source_id,
std::move(result), std::move(new_promise));
});
file_sources_[file_source_id.get()].visit(overloaded(
auto index = static_cast<size_t>(file_source_id.get()) - 1;
CHECK(index < file_sources_.size());
file_sources_[index].visit(overloaded(
[&](const FileSourceMessage &source) {
send_closure_later(G()->messages_manager(), &MessagesManager::get_message_from_server, source.full_message_id,
std::move(promise), "FileSourceMessage", nullptr);
@ -382,72 +377,6 @@ void FileReferenceManager::reload_photo(PhotoSizeSource source, Promise<Unit> pr
}
}
void FileReferenceManager::memory_cleanup() {
if (!G()->shared_config().get_option_boolean("experiment_enable_file_reference_cleanup", true)) {
return;
}
auto print_debug_messages = G()->shared_config().get_option_boolean("experiment_debug_file_reference_cleanup", false);
if (print_debug_messages) LOG(ERROR) << "memory_cleanup begin";
// Iterate all file sources and delete the unused ones
auto file_source_it = file_sources_.begin();
while (file_source_it != file_sources_.end()) {
if (print_debug_messages) LOG(ERROR) << "memory_cleanup >> file_source loop begin";
auto source_id = file_source_it->first;
// Mark immediately the file source as unused
auto file_source_unused = true;
// Iterate all the file nodes while the source is unused
auto file_nodes_it = nodes_.begin();
while (file_nodes_it != nodes_.end() && file_source_unused) {
if (print_debug_messages) LOG(ERROR) << "memory_cleanup >> file_source loop >> file_nodes loop begin";
// Get all the file sources related to the current file node
auto elements = get_all_file_sources(file_nodes_it->first);
// Iterate all the file sources related to the current file node
auto elements_it = elements.begin();
while (elements_it != elements.end() && file_source_unused) {
if (print_debug_messages) LOG(ERROR) << "memory_cleanup >> file_source loop >> file_nodes loop >> elements loop begin";
if (source_id == (u_long) elements_it->get()) {
if (print_debug_messages) LOG(ERROR) << "memory_cleanup >> file_source loop >> file_nodes loop >> elements loop break";
file_source_unused = false;
} else {
elements_it++;
if (print_debug_messages) LOG(ERROR) << "memory_cleanup >> file_source loop >> file_nodes loop >> elements loop next";
}
}
if (print_debug_messages) LOG(ERROR) << "memory_cleanup >> file_source loop >> file_nodes loop >> elements loop end";
file_nodes_it++;
if (print_debug_messages) LOG(ERROR) << "memory_cleanup >> file_source loop >> file_nodes loop next";
}
if (print_debug_messages) LOG(ERROR) << "memory_cleanup >> file_source loop >> file_nodes loop end";
if (file_source_unused) {
if (print_debug_messages) LOG(ERROR) << "memory_cleanup >> file_source loop >> remove element from file_source";
file_source_it = file_sources_.erase(file_source_it);
} else {
file_source_it++;
}
if (print_debug_messages) LOG(ERROR) << "memory_cleanup >> file_source loop next";
}
if (print_debug_messages) LOG(ERROR) << "memory_cleanup >> file_source loop end";
if (print_debug_messages) LOG(ERROR) << "memory_cleanup end";
}
void FileReferenceManager::memory_cleanup(NodeId node_id) {
auto find_node = nodes_.find(node_id);
if (find_node != nodes_.end()) {
auto &node = find_node->second;
node.query.reset();
node.file_source_ids.reset_position();
nodes_.erase(node_id);
}
}
void FileReferenceManager::memory_stats(vector<string> &output) {
output.push_back("\"nodes_\":"); output.push_back(std::to_string(nodes_.size()));
output.push_back(",");

11
td/telegram/FileReferenceManager.h
View File

@ -72,12 +72,6 @@ class FileReferenceManager final : public Actor {
template <class ParserT>
FileSourceId parse_file_source(Td *td, ParserT &parser);
void memory_cleanup(NodeId node_id);
vector<FileSourceId> get_all_file_sources(NodeId node_id);
void memory_cleanup();
void memory_stats(vector<string> &output);
private:
@ -145,10 +139,7 @@ class FileReferenceManager final : public Actor {
Variant<FileSourceMessage, FileSourceUserPhoto, FileSourceChatPhoto, FileSourceChannelPhoto, FileSourceWallpapers,
FileSourceWebPage, FileSourceSavedAnimations, FileSourceRecentStickers, FileSourceFavoriteStickers,
FileSourceBackground, FileSourceChatFull, FileSourceChannelFull>;
std::unordered_map<u_long, FileSource> file_sources_;
u_long unique_file_source_id = 0;
vector<FileSource> file_sources_;
int64 query_generation_{0};

6
td/telegram/FileReferenceManager.hpp
View File

@ -28,9 +28,9 @@ namespace td {
template <class StorerT>
void FileReferenceManager::store_file_source(FileSourceId file_source_id, StorerT &storer) const {
auto source_tuple = file_sources_.find(file_source_id.get());
auto source = source_tuple->second;
auto index = static_cast<size_t>(file_source_id.get()) - 1;
CHECK(index < file_sources_.size());
auto &source = file_sources_[index];
td::store(source.get_offset(), storer);
source.visit(overloaded([&](const FileSourceMessage &source) { td::store(source.full_message_id, storer); },
[&](const FileSourceUserPhoto &source) {

112
td/telegram/GroupCallManager.cpp
View File

@ -938,30 +938,6 @@ GroupCallManager::~GroupCallManager() = default;
void GroupCallManager::tear_down() {
parent_.reset();
if (td_->memory_manager_->can_manage_memory()) {
// Completely clear memory when closing, to avoid memory leaks
memory_cleanup(true);
}
}
void GroupCallManager::memory_cleanup() {
memory_cleanup(false);
}
void GroupCallManager::memory_cleanup(bool full) {
this->group_call_participants_.clear();
this->group_call_participants_.rehash(0);
this->group_call_recent_speakers_.clear();
this->group_call_recent_speakers_.rehash(0);
this->group_calls_.clear();
this->group_calls_.rehash(0);
//todo: check if we can clear call ids vector
// this->input_group_call_ids_.clear();
this->pending_join_requests_.clear();
this->pending_join_requests_.rehash(0);
}
void GroupCallManager::memory_stats(vector<string> &output) {
@ -1027,9 +1003,7 @@ void GroupCallManager::on_check_group_call_is_joined_timeout(GroupCallId group_c
auto input_group_call_id = get_input_group_call_id(group_call_id).move_as_ok();
auto *group_call = get_group_call(input_group_call_id);
if(!(group_call != nullptr && group_call->is_inited)) {
return;
}
CHECK(group_call != nullptr && group_call->is_inited);
auto audio_source = group_call->audio_source;
if (!group_call->is_joined || is_group_call_being_joined(input_group_call_id) ||
check_group_call_is_joined_timeout_.has_timeout(group_call_id.get()) || audio_source == 0) {
@ -1065,9 +1039,7 @@ void GroupCallManager::on_send_speaking_action_timeout(GroupCallId group_call_id
auto input_group_call_id = get_input_group_call_id(group_call_id).move_as_ok();
auto *group_call = get_group_call(input_group_call_id);
if(!(group_call != nullptr && group_call->is_inited && group_call->dialog_id.is_valid())) {
return;
}
CHECK(group_call != nullptr && group_call->is_inited && group_call->dialog_id.is_valid());
if (!group_call->is_joined || !group_call->is_speaking) {
return;
}
@ -1385,9 +1357,7 @@ void GroupCallManager::on_update_group_call_rights(InputGroupCallId input_group_
auto group_call = get_group_call(input_group_call_id);
if (need_group_call_participants(input_group_call_id, group_call)) {
if(!(group_call != nullptr && group_call->is_inited)) {
return;
}
CHECK(group_call != nullptr && group_call->is_inited);
try_load_group_call_administrators(input_group_call_id, group_call->dialog_id);
auto *group_call_participants = add_group_call_participants(input_group_call_id);
@ -1468,9 +1438,7 @@ void GroupCallManager::finish_get_group_call(InputGroupCallId input_group_call_i
}
auto group_call = get_group_call(input_group_call_id);
if(!(group_call != nullptr && group_call->is_inited)) {
return;
}
CHECK(group_call != nullptr && group_call->is_inited);
for (auto &promise : promises) {
if (promise) {
promise.set_value(get_group_call_object(group_call, get_recent_speakers(group_call, false)));
@ -1494,12 +1462,8 @@ void GroupCallManager::finish_check_group_call_is_joined(InputGroupCallId input_
}
auto *group_call = get_group_call(input_group_call_id);
if(!(group_call != nullptr && group_call->is_inited)) {
return;
}
if(!(audio_source != 0)) {
return;
}
CHECK(group_call != nullptr && group_call->is_inited);
CHECK(audio_source != 0);
if (!group_call->is_joined || is_group_call_being_joined(input_group_call_id) ||
check_group_call_is_joined_timeout_.has_timeout(group_call->group_call_id.get()) ||
group_call->audio_source != audio_source) {
@ -1601,9 +1565,7 @@ void GroupCallManager::on_get_group_call_participants(
bool is_sync = is_load && offset.empty();
if (is_sync) {
auto group_call = get_group_call(input_group_call_id);
if(!(group_call != nullptr && group_call->is_inited)) {
return;
}
CHECK(group_call != nullptr && group_call->is_inited);
is_sync = group_call->syncing_participants;
if (is_sync) {
group_call->syncing_participants = false;
@ -1637,9 +1599,7 @@ void GroupCallManager::on_get_group_call_participants(
if (is_empty || is_sync) {
bool need_update = false;
auto group_call = get_group_call(input_group_call_id);
if(!(group_call != nullptr && group_call->is_inited)) {
return;
}
CHECK(group_call != nullptr && group_call->is_inited);
if (is_empty && !group_call->loaded_all_participants) {
group_call->loaded_all_participants = true;
need_update = true;
@ -1736,13 +1696,7 @@ void GroupCallManager::on_update_group_call_participants(
int32 video_diff = 0;
bool need_update = false;
auto group_call = get_group_call(input_group_call_id);
if (group_call == nullptr) {
return;
}
for (auto &group_call_participant : participants) {
if (group_call_participant == nullptr) {
continue;
}
GroupCallParticipant participant(group_call_participant, version);
if (!participant.is_valid()) {
LOG(ERROR) << "Receive invalid " << to_string(group_call_participant);
@ -1856,9 +1810,7 @@ bool GroupCallManager::process_pending_group_call_participant_updates(InputGroup
return false;
}
auto group_call = get_group_call(input_group_call_id);
if(!(group_call != nullptr && group_call->is_inited)) {
return false;
}
CHECK(group_call != nullptr && group_call->is_inited);
if (group_call->version == -1 || !group_call->is_active) {
return false;
}
@ -1972,9 +1924,7 @@ void GroupCallManager::sync_group_call_participants(InputGroupCallId input_group
}
auto group_call = get_group_call(input_group_call_id);
if (!(group_call != nullptr && group_call->is_inited)) {
return;
}
CHECK(group_call != nullptr && group_call->is_inited);
sync_participants_timeout_.cancel_timeout(group_call->group_call_id.get());
@ -2002,12 +1952,8 @@ void GroupCallManager::on_sync_group_call_participants(InputGroupCallId input_gr
if (result.is_error()) {
auto group_call = get_group_call(input_group_call_id);
if (!(group_call != nullptr && group_call->is_inited)) {
return;
}
if (!(group_call->syncing_participants)) {
return;
}
CHECK(group_call != nullptr && group_call->is_inited);
CHECK(group_call->syncing_participants);
group_call->syncing_participants = false;
sync_participants_timeout_.add_timeout_in(group_call->group_call_id.get(),
@ -2239,9 +2185,7 @@ std::pair<int32, int32> GroupCallManager::process_group_call_participant(InputGr
if (participant.is_self) {
auto *group_call = get_group_call(input_group_call_id);
if (!(group_call != nullptr && group_call->is_inited)) {
return {0, 0};
}
CHECK(group_call != nullptr && group_call->is_inited);
auto can_self_unmute = group_call->is_active && !participant.get_is_muted_by_admin();
if (can_self_unmute != group_call->can_self_unmute) {
group_call->can_self_unmute = can_self_unmute;
@ -3994,9 +3938,7 @@ void GroupCallManager::load_group_call_participants(GroupCallId group_call_id, i
if (!need_group_call_participants(input_group_call_id, group_call)) {
return promise.set_error(Status::Error(400, "Can't load group call participants"));
}
if (!(group_call != nullptr && group_call->is_inited)) {
return promise.set_error(Status::Error(400, "Internal error"));
}
CHECK(group_call != nullptr && group_call->is_inited);
if (group_call->loaded_all_participants) {
return promise.set_value(Unit());
}
@ -4066,9 +4008,7 @@ void GroupCallManager::on_group_call_left(InputGroupCallId input_group_call_id,
}
auto *group_call = get_group_call(input_group_call_id);
if (!(group_call != nullptr && group_call->is_inited)) {
return;
}
CHECK(group_call != nullptr && group_call->is_inited);
if (group_call->is_joined && group_call->audio_source == audio_source) {
on_group_call_left_impl(group_call, need_rejoin, "on_group_call_left");
send_update_group_call(group_call, "on_group_call_left");
@ -4076,9 +4016,9 @@ void GroupCallManager::on_group_call_left(InputGroupCallId input_group_call_id,
}
void GroupCallManager::on_group_call_left_impl(GroupCall *group_call, bool need_rejoin, const char *source) {
if (!(group_call != nullptr && group_call->is_inited && group_call->is_joined)) {
return;
}
CHECK(group_call != nullptr && group_call->is_inited && group_call->is_joined);
LOG(INFO) << "Leave " << group_call->group_call_id << " in " << group_call->dialog_id
<< " with need_rejoin = " << need_rejoin << " from " << source;
group_call->is_joined = false;
group_call->need_rejoin = need_rejoin && !group_call->is_being_left;
if (group_call->need_rejoin && group_call->dialog_id.is_valid()) {
@ -4156,9 +4096,7 @@ bool GroupCallManager::try_clear_group_call_participants(InputGroupCallId input_
CHECK(participants != nullptr);
group_call_participants_.erase(participants_it);
if (!(group_call != nullptr && group_call->is_inited)) {
return false;
}
CHECK(group_call != nullptr && group_call->is_inited);
LOG(INFO) << "Clear participants in " << input_group_call_id << " from " << group_call->dialog_id;
if (group_call->loaded_all_participants) {
group_call->loaded_all_participants = false;
@ -4441,9 +4379,7 @@ void GroupCallManager::on_receive_group_call_version(InputGroupCallId input_grou
if (!need_group_call_participants(input_group_call_id, group_call)) {
return;
}
if (!(group_call != nullptr && group_call->is_inited)) {
return;
}
CHECK(group_call != nullptr && group_call->is_inited);
if (group_call->version == -1) {
return;
}
@ -4726,9 +4662,7 @@ void GroupCallManager::update_group_call_dialog(const GroupCall *group_call, con
vector<td_api::object_ptr<td_api::groupCallRecentSpeaker>> GroupCallManager::get_recent_speakers(
const GroupCall *group_call, bool for_update) {
if(!(group_call != nullptr && group_call->is_inited)) {
return Auto();
}
CHECK(group_call != nullptr && group_call->is_inited);
auto recent_speakers_it = group_call_recent_speakers_.find(group_call->group_call_id);
if (recent_speakers_it == group_call_recent_speakers_.end()) {
@ -4841,9 +4775,7 @@ void GroupCallManager::send_update_group_call_participant(InputGroupCallId input
return;
}
auto group_call = get_group_call(input_group_call_id);
if(!(group_call != nullptr && group_call->is_inited)) {
return;
}
CHECK(group_call != nullptr && group_call->is_inited);
send_update_group_call_participant(group_call->group_call_id, participant, source);
}

4
td/telegram/GroupCallManager.h
View File

@ -39,8 +39,6 @@ class GroupCallManager final : public Actor {
GroupCallManager &operator=(GroupCallManager &&) = delete;
~GroupCallManager() final;
void memory_cleanup();
void memory_stats(vector<string> &output);
DialogId get_group_call_participant_id(const td_api::object_ptr<td_api::MessageSender> &message_sender);
@ -157,8 +155,6 @@ class GroupCallManager final : public Actor {
void tear_down() final;
void memory_cleanup(bool full);
static void on_update_group_call_participant_order_timeout_callback(void *group_call_manager_ptr,
int64 group_call_id_int);

18
td/telegram/InlineQueriesManager.cpp
View File

@ -174,10 +174,6 @@ InlineQueriesManager::InlineQueriesManager(Td *td, ActorShared<> parent) : td_(t
void InlineQueriesManager::tear_down() {
parent_.reset();
if (td_->memory_manager_->can_manage_memory()) {
// Completely clear memory when closing, to avoid memory leaks
memory_cleanup(true);
}
}
void InlineQueriesManager::on_drop_inline_query_result_timeout_callback(void *inline_queries_manager_ptr,
@ -1906,20 +1902,6 @@ void InlineQueriesManager::remove_recent_inline_bot(UserId bot_user_id, Promise<
promise.set_value(Unit());
}
void InlineQueriesManager::memory_cleanup() {
memory_cleanup(false);
}
void InlineQueriesManager::memory_cleanup(bool full) {
recently_used_bot_user_ids_.clear();
inline_query_results_.clear();
inline_query_results_.rehash(0);
inline_message_contents_.clear();
inline_message_contents_.rehash(0);
query_id_to_bot_user_id_.clear();
query_id_to_bot_user_id_.rehash(0);
}
void InlineQueriesManager::memory_stats(vector<string> &output) {
output.push_back("\"recently_used_bot_user_ids_\":"); output.push_back(std::to_string(recently_used_bot_user_ids_.size()));
output.push_back(",");

4
td/telegram/InlineQueriesManager.h
View File

@ -38,8 +38,6 @@ class InlineQueriesManager final : public Actor {
public:
InlineQueriesManager(Td *td, ActorShared<> parent);
void memory_cleanup();
void memory_stats(vector<string> &output);
void after_get_difference();
@ -118,8 +116,6 @@ class InlineQueriesManager final : public Actor {
void tear_down() final;
void memory_cleanup(bool full);
int32 recently_used_bots_loaded_ = 0; // 0 - not loaded, 1 - load request was sent, 2 - loaded
MultiPromiseActor resolve_recent_inline_bots_multipromise_{"ResolveRecentInlineBotsMultiPromiseActor"};

5
td/telegram/Log.cpp
View File

@ -70,9 +70,4 @@ void Log::set_fatal_error_callback(FatalErrorCallbackPtr callback) {
}
}
void Log::set_disable_death_handler(bool disabled) {
std::lock_guard<std::mutex> lock(log_mutex);
set_log_disable_death_handler(disabled);
}
} // namespace td

2
td/telegram/Log.h
View File

@ -81,8 +81,6 @@ class Log {
* Pass nullptr to remove the callback.
*/
static void set_fatal_error_callback(FatalErrorCallbackPtr callback);
static void set_disable_death_handler(bool disabled);
};
} // namespace td

3
td/telegram/Logging.cpp
View File

@ -57,8 +57,7 @@ static const std::map<Slice, int *> log_tags{
ADD_TAG(proxy), ADD_TAG(net_query), ADD_TAG(td_requests), ADD_TAG(dc),
ADD_TAG(file_loader), ADD_TAG(mtproto), ADD_TAG(raw_mtproto), ADD_TAG(fd),
ADD_TAG(actor), ADD_TAG(sqlite), ADD_TAG(notifications), ADD_TAG(get_difference),
ADD_TAG(file_gc), ADD_TAG(config_recoverer), ADD_TAG(dns_resolver), ADD_TAG(file_references),
ADD_TAG(messages), ADD_TAG(postponed_pts_update), ADD_TAG(add_pending_update)};
ADD_TAG(file_gc), ADD_TAG(config_recoverer), ADD_TAG(dns_resolver), ADD_TAG(file_references)};
#undef ADD_TAG
Status Logging::set_current_stream(td_api::object_ptr<td_api::LogStream> stream) {

64
td/telegram/MemoryManager.cpp
View File

@ -75,40 +75,13 @@ MemoryManager::MemoryManager(Td *td, ActorShared<> parent) : td_(td), parent_(st
}
void MemoryManager::start_up() {
LOG(INFO) << "Start MemoryManager";
}
void MemoryManager::tear_down() {
LOG(INFO) << "Stopping MemoryManager";
parent_.reset();
LOG(INFO) << "Stopped MemoryManager";
}
bool MemoryManager::can_manage_memory() const {
if (!(td_->auth_manager_->is_authorized() && !G()->close_flag())) {
return false;
}
if (!do_session_settings_allow_for_memory_management()) {
return false;
}
return true;
}
bool MemoryManager::do_session_settings_allow_for_memory_management() {
if (G()->parameters().use_message_db || G()->parameters().use_chat_info_db || G()->parameters().use_file_db) {
return false;
}
return true;
}
void MemoryManager::get_memory_stats(bool full, Promise<MemoryStats> promise) const {
if (!can_manage_memory()) {
auto value = MemoryStats(R"({"warning": "OptimizeMemory is not enabled, please read the recommended options on README.md"})");
promise.set_value(std::move(value));
return;
}
vector<string> output = {"{"};
output.push_back("\"memory_stats\":{");
@ -213,43 +186,6 @@ void MemoryManager::get_memory_stats(bool full, Promise<MemoryStats> promise) co
promise.set_value(std::move(value));
}
void MemoryManager::clean_memory(bool full, Promise<Unit> promise) const {
if (!can_manage_memory()) {
if (!do_session_settings_allow_for_memory_management()) {
promise.set_error(Status::Error(405, "MEMORY_STATS_DISALLOWED"
" Session settings don't allow memory optimization."
" If you want to optimize memory, you should completely disable all databases."));
} else {
promise.set_error(Status::Error(405, "Can't manage memory now"));
}
return;
}
td_->messages_manager_->memory_cleanup();
td_->contacts_manager_->memory_cleanup();
td_->web_pages_manager_->memory_cleanup();
td_->stickers_manager_->memory_cleanup();
td_->documents_manager_->memory_cleanup();
td_->video_notes_manager_->memory_cleanup();
td_->videos_manager_->memory_cleanup();
td_->audios_manager_->memory_cleanup();
td_->animations_manager_->memory_cleanup();
td_->file_manager_->memory_cleanup();
td_->file_reference_manager_->memory_cleanup();
td_->group_call_manager_->memory_cleanup();
td_->background_manager_->memory_cleanup();
td_->inline_queries_manager_->memory_cleanup();
td_->poll_manager_->memory_cleanup();
#ifdef __linux__
#if defined(__GLIBC__) && !defined(__UCLIBC__) && !defined(__MUSL__)
malloc_trim(0);
#endif
#endif
promise.set_value(Unit());
}
void MemoryManager::get_current_state(vector<td_api::object_ptr<td_api::Update>> &updates) const {
if (td_->auth_manager_->is_bot()) {
return;

6
td/telegram/MemoryManager.h
View File

@ -48,14 +48,8 @@ class MemoryManager : public Actor {
public:
MemoryManager(Td *td, ActorShared<> parent);
bool can_manage_memory() const;
static bool do_session_settings_allow_for_memory_management();
void get_memory_stats(bool full, Promise<MemoryStats> promise) const;
void clean_memory(bool full, Promise<Unit> promise) const;
void get_current_state(vector<td_api::object_ptr<td_api::Update>> &updates) const;
private:

713
td/telegram/MessagesManager.cpp
View File

File diff suppressed because it is too large Load Diff

32
td/telegram/MessagesManager.h
View File

@ -141,8 +141,6 @@ class MessagesManager final : public Actor {
MessagesManager &operator=(MessagesManager &&) = delete;
~MessagesManager() final;
void memory_cleanup();
void memory_stats(vector<string> &output);
td_api::object_ptr<td_api::MessageSender> get_message_sender_object_const(UserId user_id, DialogId dialog_id,
@ -1147,9 +1145,6 @@ class MessagesManager final : public Actor {
struct Dialog {
DialogId dialog_id;
int64 time_ = INT64_MAX;
MessageId last_new_message_id; // identifier of the last known server message received from update, there should be
// no server messages after it
MessageId last_message_id; // identifier of the message after which currently there are no messages, i.e. a
@ -1357,18 +1352,6 @@ class MessagesManager final : public Actor {
Dialog &operator=(Dialog &&other) = delete;
~Dialog();
void set_time() {
time_ = std::time(nullptr);
}
int64 get_time() const {
return time_;
}
void reset_time() {
time_ = INT64_MAX;
}
template <class StorerT>
void store(StorerT &storer) const;
@ -1729,10 +1712,6 @@ class MessagesManager final : public Actor {
static constexpr bool DROP_SEND_MESSAGE_UPDATES = false;
void memory_cleanup(bool teardown);
void memory_cleanup(DialogId dialog_id, Dialog *d);
static FullMessageId get_full_message_id(const tl_object_ptr<telegram_api::Message> &message_ptr, bool is_scheduled);
static int32 get_message_date(const tl_object_ptr<telegram_api::Message> &message_ptr);
@ -2910,9 +2889,6 @@ class MessagesManager final : public Actor {
void on_channel_get_difference_timeout(DialogId dialog_id);
void get_channel_difference_delayed(DialogId dialog_id, int32 pts, bool force, bool enable_reactive_channel_difference,
const char *source);
void get_channel_difference(DialogId dialog_id, int32 pts, bool force, const char *source);
void do_get_channel_difference(DialogId dialog_id, int32 pts, bool force,
@ -3244,14 +3220,6 @@ class MessagesManager final : public Actor {
};
std::unordered_map<int64, unique_ptr<PendingMessageImport>> pending_message_imports_;
struct PendingChannelDifference {
DialogId dialog_id;
int32 pts;
bool force;
};
std::unordered_map<int64, PendingChannelDifference> pending_channel_difference_;
int64 last_pending_channel_difference_ = 0;
struct PendingMessageGroupSend {
DialogId dialog_id;
size_t finished_count = 0;

3
td/telegram/Photo.cpp
View File

@ -461,9 +461,6 @@ Variant<PhotoSize, string> get_photo_size(FileManager *file_manager, PhotoSizeSo
AnimationSize get_animation_size(FileManager *file_manager, PhotoSizeSource source, int64 id, int64 access_hash,
std::string file_reference, DcId dc_id, DialogId owner_dialog_id,
tl_object_ptr<telegram_api::videoSize> &&size) {
if (size == nullptr) {
return {};
}
CHECK(size != nullptr);
AnimationSize res;
if (size->type_ != "v" && size->type_ != "u") {

87
td/telegram/PollManager.cpp
View File

@ -268,10 +268,6 @@ void PollManager::start_up() {
void PollManager::tear_down() {
parent_.reset();
if (td_->memory_manager_->can_manage_memory()) {
// Completely clear memory when closing, to avoid memory leaks
memory_cleanup(true);
}
}
PollManager::~PollManager() = default;
@ -344,7 +340,7 @@ void PollManager::save_poll(const Poll *poll, PollId poll_id) {
}
LOG(INFO) << "Save " << poll_id << " to database";
if (!(poll != nullptr)) return;
CHECK(poll != nullptr);
G()->td_db()->get_sqlite_pmc()->set(get_poll_database_key(poll_id), log_event_store(*poll).as_slice().str(), Auto());
}
@ -502,19 +498,7 @@ vector<int32> PollManager::get_vote_percentage(const vector<int32> &voter_counts
td_api::object_ptr<td_api::poll> PollManager::get_poll_object(PollId poll_id) const {
auto poll = get_poll(poll_id);
if (!(poll != nullptr)) {
//todo: find better alternative, rather than just creating a fake poll to avoid crashes...
vector<td_api::object_ptr<td_api::pollOption>> poll_options;
poll_options.push_back(td_api::make_object<td_api::pollOption>(
"empty", 0, 0, false,
false));
vector<int32> votes;
return td_api::make_object<td_api::poll>(
poll_id.get(), "empty question", std::move(poll_options), 0,
std::move(votes), true,
td_api::make_object<td_api::pollTypeRegular>(false),
1, G()->unix_time() - 1, true);
}
CHECK(poll != nullptr);
return get_poll_object(poll_id, poll);
}
@ -638,9 +622,7 @@ PollId PollManager::create_poll(string &&question, vector<string> &&options, boo
}
void PollManager::register_poll(PollId poll_id, FullMessageId full_message_id, const char *source) {
if(!(have_poll(poll_id))) {
return;
}
CHECK(have_poll(poll_id));
if (full_message_id.get_message_id().is_scheduled()) {
return;
}
@ -651,7 +633,7 @@ void PollManager::register_poll(PollId poll_id, FullMessageId full_message_id, c
bool is_inserted = poll_messages_[poll_id].insert(full_message_id).second;
LOG_CHECK(is_inserted) << source << " " << poll_id << " " << full_message_id;
auto poll = get_poll(poll_id);
if (!(poll != nullptr)) return;
CHECK(poll != nullptr);
if (!td_->auth_manager_->is_bot() && !is_local_poll_id(poll_id) &&
!(poll->is_closed && poll->is_updated_after_close)) {
update_poll_timeout_.add_timeout_in(poll_id.get(), 0);
@ -659,9 +641,7 @@ void PollManager::register_poll(PollId poll_id, FullMessageId full_message_id, c
}
void PollManager::unregister_poll(PollId poll_id, FullMessageId full_message_id, const char *source) {
if (!(have_poll(poll_id))) {
return;
}
CHECK(have_poll(poll_id));
if (full_message_id.get_message_id().is_scheduled()) {
return;
}
@ -671,10 +651,7 @@ void PollManager::unregister_poll(PollId poll_id, FullMessageId full_message_id,
LOG(INFO) << "Unregister " << poll_id << " from " << full_message_id << " from " << source;
auto &message_ids = poll_messages_[poll_id];
auto is_deleted = message_ids.erase(full_message_id) > 0;
if (!is_deleted) {
LOG(DEBUG) << "tried to unregister already deleted poll. " << source << " " << poll_id << " " << full_message_id;
return;
}
LOG_CHECK(is_deleted) << source << " " << poll_id << " " << full_message_id;
if (message_ids.empty()) {
poll_messages_.erase(poll_id);
update_poll_timeout_.cancel_timeout(poll_id.get());
@ -714,7 +691,7 @@ void PollManager::set_poll_answer(PollId poll_id, FullMessageId full_message_id,
}
auto poll = get_poll(poll_id);
if (!(poll != nullptr)) return promise.set_error(Status::Error(400, "Poll can't be answered"));
CHECK(poll != nullptr);
if (poll->is_closed) {
return promise.set_error(Status::Error(400, "Can't answer closed poll"));
}
@ -963,7 +940,7 @@ void PollManager::get_poll_voters(PollId poll_id, FullMessageId full_message_id,
}
auto poll = get_poll(poll_id);
if (!(poll != nullptr)) return promise.set_error(Status::Error(400, "Poll results can't be received"));
CHECK(poll != nullptr);
if (option_id < 0 || static_cast<size_t>(option_id) >= poll->options.size()) {
return promise.set_error(Status::Error(400, "Invalid option ID specified"));
}
@ -1013,7 +990,7 @@ void PollManager::get_poll_voters(PollId poll_id, FullMessageId full_message_id,
void PollManager::on_get_poll_voters(PollId poll_id, int32 option_id, string offset, int32 limit,
Result<tl_object_ptr<telegram_api::messages_votesList>> &&result) {
auto poll = get_poll(poll_id);
if (!(poll != nullptr)) return;
CHECK(poll != nullptr);
if (option_id < 0 || static_cast<size_t>(option_id) >= poll->options.size()) {
LOG(ERROR) << "Can't process voters for option " << option_id << " in " << poll_id << ", because it has only "
<< poll->options.size() << " options";
@ -1107,10 +1084,7 @@ void PollManager::stop_poll(PollId poll_id, FullMessageId full_message_id, uniqu
return;
}
auto poll = get_poll_editable(poll_id);
if (!(poll != nullptr)) {
promise.set_value(Unit());
return;
}
CHECK(poll != nullptr);
if (poll->is_closed) {
promise.set_value(Unit());
return;
@ -1163,7 +1137,7 @@ void PollManager::do_stop_poll(PollId poll_id, FullMessageId full_message_id, un
void PollManager::stop_local_poll(PollId poll_id) {
CHECK(is_local_poll_id(poll_id));
auto poll = get_poll_editable(poll_id);
if (!(poll != nullptr)) return;
CHECK(poll != nullptr);
if (poll->is_closed) {
return;
}
@ -1184,12 +1158,8 @@ void PollManager::on_update_poll_timeout(PollId poll_id) {
if (G()->close_flag()) {
return;
}
auto poll = get_poll(poll_id);
if (poll == nullptr) {
return;
}
CHECK(poll != nullptr);
if (poll->is_closed && poll->is_updated_after_close) {
return;
}
@ -1203,13 +1173,7 @@ void PollManager::on_update_poll_timeout(PollId poll_id) {
return;
}
auto full_message_id_set = std::move(it->second);
if (full_message_id_set.empty()) {
return;
}
auto full_message_id = *full_message_id_set.begin();
auto full_message_id = *it->second.begin();
LOG(INFO) << "Fetching results of " << poll_id << " from " << full_message_id;
auto query_promise = PromiseCreator::lambda([poll_id, generation = current_generation_, actor_id = actor_id(this)](
Result<tl_object_ptr<telegram_api::Updates>> &&result) {
@ -1226,7 +1190,7 @@ void PollManager::on_close_poll_timeout(PollId poll_id) {
}
auto poll = get_poll_editable(poll_id);
if (!(poll != nullptr)) return;
CHECK(poll != nullptr);
if (poll->is_closed || poll->close_date == 0) {
return;
}
@ -1250,7 +1214,7 @@ void PollManager::on_close_poll_timeout(PollId poll_id) {
void PollManager::on_get_poll_results(PollId poll_id, uint64 generation,
Result<tl_object_ptr<telegram_api::Updates>> result) {
auto poll = get_poll(poll_id);
if (!(poll != nullptr)) return;
CHECK(poll != nullptr);
if (result.is_error()) {
if (!(poll->is_closed && poll->is_updated_after_close) && !G()->close_flag() && !td_->auth_manager_->is_bot()) {
auto timeout = get_polling_timeout();
@ -1302,7 +1266,7 @@ PollId PollManager::dup_poll(PollId poll_id) {
bool PollManager::has_input_media(PollId poll_id) const {
auto poll = get_poll(poll_id);
if (!(poll != nullptr)) return false;
CHECK(poll != nullptr);
return !poll->is_quiz || poll->correct_option_id >= 0;
}
@ -1412,7 +1376,7 @@ PollId PollManager::on_get_poll(PollId poll_id, tl_object_ptr<telegram_api::poll
bool is_inserted = polls_.emplace(poll_id, std::move(p)).second;
CHECK(is_inserted);
}
if (!(poll != nullptr)) return PollId();
CHECK(poll != nullptr);
bool poll_server_is_closed = false;
if (poll_server != nullptr) {
@ -1753,23 +1717,6 @@ void PollManager::on_binlog_events(vector<BinlogEvent> &&events) {
}
}
void PollManager::memory_cleanup() {
memory_cleanup(false);
}
void PollManager::memory_cleanup(bool full) {
polls_.clear();
polls_.rehash(0);
poll_messages_.clear();
poll_messages_.rehash(0);
poll_voters_.clear();
poll_voters_.rehash(0);
loaded_from_database_polls_.clear();
loaded_from_database_polls_.rehash(0);
being_closed_polls_.clear();
being_closed_polls_.rehash(0);
}
void PollManager::memory_stats(vector<string> &output) {
output.push_back("\"polls_\":"); output.push_back(std::to_string(polls_.size()));
output.push_back(",");

4
td/telegram/PollManager.h
View File

@ -43,8 +43,6 @@ class PollManager final : public Actor {
PollManager &operator=(PollManager &&) = delete;
~PollManager() final;
void memory_cleanup();
void memory_stats(vector<string> &output);
static bool is_local_poll_id(PollId poll_id);
@ -147,8 +145,6 @@ class PollManager final : public Actor {
void start_up() final;
void tear_down() final;
void memory_cleanup(bool full);
static void on_update_poll_timeout_callback(void *poll_manager_ptr, int64 poll_id_int);
static void on_close_poll_timeout_callback(void *poll_manager_ptr, int64 poll_id_int);

4
td/telegram/PollManager.hpp
View File

@ -128,9 +128,7 @@ void PollManager::store_poll(PollId poll_id, StorerT &storer) const {
td::store(poll_id.get(), storer);
if (is_local_poll_id(poll_id)) {
auto poll = get_poll(poll_id);
if (poll == nullptr) {
return;
}
CHECK(poll != nullptr);
bool has_open_period = poll->open_period != 0;
bool has_close_date = poll->close_date != 0;
bool has_explanation = !poll->explanation.text.empty();

8
td/telegram/SequenceDispatcher.cpp
View File

@ -257,17 +257,13 @@ void MultiSequenceDispatcher::send_with_callback(NetQueryPtr query, ActorShared<
void MultiSequenceDispatcher::on_result() {
auto it = dispatchers_.find(get_link_token());
if (it == dispatchers_.end()) {
return;
}
CHECK(it != dispatchers_.end());
it->second.cnt_--;
}
void MultiSequenceDispatcher::ready_to_close() {
auto it = dispatchers_.find(get_link_token());
if (it == dispatchers_.end()) {
return;
}
CHECK(it != dispatchers_.end());
if (it->second.cnt_ == 0) {
LOG(DEBUG) << "Close SequenceDispatcher " << get_link_token();
dispatchers_.erase(it);

18
td/telegram/SetWithPosition.h
View File

@ -18,14 +18,6 @@ namespace td {
template <class T>
class FastSetWithPosition {
public:
std::vector<T> get_unchecked() const {
std::vector<T> res;
res.insert(res.end(), checked_.begin(), checked_.end());
res.insert(res.end(), not_checked_.begin(), not_checked_.end());
td::unique(res);
return res;
}
std::vector<T> get_some_elements() const {
std::vector<T> res;
res.reserve(4);
@ -114,16 +106,6 @@ class FastSetWithPosition {
template <class T>
class SetWithPosition {
public:
std::vector<T> get_all_elements() const {
if (fast_) {
return fast_->get_unchecked();
}
if (has_value_) {
return {value_};
}
return {};
}
std::vector<T> get_some_elements() const {
if (fast_) {
return fast_->get_some_elements();

87
td/telegram/StickersManager.cpp
View File

@ -1401,9 +1401,7 @@ void StickersManager::on_load_special_sticker_set(const SpecialStickerSetType &t
CHECK(special_sticker_set.id_.is_valid());
auto sticker_set = get_sticker_set(special_sticker_set.id_);
if (sticker_set == nullptr) {
return;
}
CHECK(sticker_set != nullptr);
CHECK(sticker_set->was_loaded);
if (type.type_ == SpecialStickerSetType::animated_emoji_click()) {
@ -1442,8 +1440,6 @@ void StickersManager::on_load_special_sticker_set(const SpecialStickerSetType &t
void StickersManager::tear_down() {
parent_.reset();
// Completely clear memory when closing, to avoid memory leaks
memory_cleanup();
}
tl_object_ptr<td_api::MaskPoint> StickersManager::get_mask_point_object(int32 point) {
@ -1705,17 +1701,9 @@ tl_object_ptr<td_api::sticker> StickersManager::get_sticker_object(FileId file_i
}
auto it = stickers_.find(file_id);
if (it == stickers_.end() || it->second == nullptr) {
return nullptr;
}
CHECK(it != stickers_.end());
auto sticker = it->second.get();
if (sticker == nullptr) {
return nullptr;
}
CHECK(sticker != nullptr);
auto mask_position = sticker->point >= 0
? make_tl_object<td_api::maskPosition>(get_mask_point_object(sticker->point),
sticker->x_shift, sticker->y_shift, sticker->scale)
@ -1778,9 +1766,7 @@ tl_object_ptr<td_api::DiceStickers> StickersManager::get_dice_stickers_object(co
}
auto sticker_set = get_sticker_set(sticker_set_id);
if (sticker_set == nullptr) {
return nullptr;
}
CHECK(sticker_set != nullptr);
if (!sticker_set->was_loaded) {
return nullptr;
}
@ -2056,8 +2042,8 @@ std::pair<int64, FileId> StickersManager::on_get_sticker_document(
StickersManager::Sticker *StickersManager::get_sticker(FileId file_id) {
auto sticker = stickers_.find(file_id);
if (sticker == stickers_.end() || sticker->second == nullptr) {
return make_unique<Sticker>().get();
if (sticker == stickers_.end()) {
return nullptr;
}
CHECK(sticker->second->file_id == file_id);
@ -2066,22 +2052,18 @@ StickersManager::Sticker *StickersManager::get_sticker(FileId file_id) {
const StickersManager::Sticker *StickersManager::get_sticker(FileId file_id) const {
auto sticker = stickers_.find(file_id);
if (sticker == stickers_.end() ||
sticker->second == nullptr ||
sticker->second->file_id != file_id) {
return make_unique<Sticker>().get();
if (sticker == stickers_.end()) {
return nullptr;
}
CHECK(sticker->second->file_id == file_id);
return sticker->second.get();
}
StickersManager::StickerSet *StickersManager::get_sticker_set(StickerSetId sticker_set_id) {
auto sticker_set = sticker_sets_.find(sticker_set_id);
if (sticker_set == sticker_sets_.end() ||
sticker_set->second == nullptr) {
return make_unique<StickerSet>().get();
if (sticker_set == sticker_sets_.end()) {
return nullptr;
}
return sticker_set->second.get();
@ -2090,7 +2072,7 @@ StickersManager::StickerSet *StickersManager::get_sticker_set(StickerSetId stick
const StickersManager::StickerSet *StickersManager::get_sticker_set(StickerSetId sticker_set_id) const {
auto sticker_set = sticker_sets_.find(sticker_set_id);
if (sticker_set == sticker_sets_.end()) {
return make_unique<StickerSet>().get();
return nullptr;
}
return sticker_set->second.get();
@ -2184,11 +2166,7 @@ void StickersManager::delete_sticker_thumbnail(FileId file_id) {
vector<FileId> StickersManager::get_sticker_file_ids(FileId file_id) const {
vector<FileId> result;
auto sticker = get_sticker(file_id);
if (sticker == nullptr) {
return result;
}
CHECK(sticker != nullptr);
result.push_back(file_id);
if (sticker->s_thumbnail.file_id.is_valid()) {
result.push_back(sticker->s_thumbnail.file_id);
@ -2220,7 +2198,7 @@ void StickersManager::merge_stickers(FileId new_id, FileId old_id, bool can_dele
CHECK(old_ != nullptr);
auto new_it = stickers_.find(new_id);
if (new_it == stickers_.end() || new_it->second == nullptr) {
if (new_it == stickers_.end()) {
auto &old = stickers_[old_id];
if (!can_delete_old) {
dup_sticker(new_id, old_id);
@ -2405,9 +2383,6 @@ bool StickersManager::has_input_media(FileId sticker_file_id, bool is_secret) co
auto file_view = td_->file_manager_->get_file_view(sticker_file_id);
if (is_secret) {
const Sticker *sticker = get_sticker(sticker_file_id);
if (sticker == nullptr) {
return false;
}
CHECK(sticker != nullptr);
if (file_view.is_encrypted_secret()) {
if (!file_view.encryption_key().empty() && file_view.has_remote_location() &&
@ -2442,11 +2417,7 @@ SecretInputMedia StickersManager::get_secret_input_media(FileId sticker_file_id,
tl_object_ptr<telegram_api::InputEncryptedFile> input_file,
BufferSlice thumbnail) const {
const Sticker *sticker = get_sticker(sticker_file_id);
if (sticker == nullptr) {
return {};
}
CHECK(sticker != nullptr);
auto file_view = td_->file_manager_->get_file_view(sticker_file_id);
if (file_view.is_encrypted_secret()) {
if (file_view.has_remote_location()) {
@ -4041,13 +4012,7 @@ void StickersManager::unregister_dice(const string &emoji, int32 value, FullMess
<< source;
auto &message_ids = dice_messages_[emoji];
auto is_deleted = message_ids.erase(full_message_id) > 0;
// Start custom-patches
if (is_deleted) {
// End custom-patches
LOG_CHECK(is_deleted) << source << " " << emoji << " " << value << " " << full_message_id;
// Start custom-patches
}
// End custom-patches
if (message_ids.empty()) {
dice_messages_.erase(emoji);
@ -6009,9 +5974,7 @@ int64 StickersManager::get_recent_stickers_hash(const vector<FileId> &sticker_id
numbers.reserve(sticker_ids.size());
for (auto sticker_id : sticker_ids) {
auto sticker = get_sticker(sticker_id);
if (sticker == nullptr) {
continue;
}
CHECK(sticker != nullptr);
auto file_view = td_->file_manager_->get_file_view(sticker_id);
CHECK(file_view.has_remote_location());
if (!file_view.remote_location().is_document()) {
@ -7156,24 +7119,6 @@ void StickersManager::get_current_state(vector<td_api::object_ptr<td_api::Update
}
}
void StickersManager::memory_cleanup() {
stickers_.clear();
stickers_.rehash(0);
sticker_sets_.clear();
sticker_sets_.rehash(0);
short_name_to_sticker_set_id_.clear();
short_name_to_sticker_set_id_.rehash(0);
attached_sticker_sets_.clear();
attached_sticker_sets_.rehash(0);
found_stickers_.clear();
found_stickers_.rehash(0);
found_sticker_sets_.clear();
found_sticker_sets_.rehash(0);
special_sticker_sets_.clear();
special_sticker_sets_.rehash(0);
dice_messages_.clear();
dice_messages_.rehash(0);
}
void StickersManager::memory_stats(vector<string> &output) {
output.push_back("\"found_stickers_\":"); output.push_back(std::to_string(found_stickers_.size()));
output.push_back(",");

2
td/telegram/StickersManager.h
View File

@ -41,8 +41,6 @@ class StickersManager final : public Actor {
public:
static constexpr int64 GREAT_MINDS_SET_ID = 1842540969984001;
void memory_cleanup();
void memory_stats(vector<string> &output);
static vector<StickerSetId> convert_sticker_set_ids(const vector<int64> &sticker_set_ids);

19
td/telegram/StickersManager.hpp
View File

@ -24,9 +24,7 @@ namespace td {
template <class StorerT>
void StickersManager::store_sticker(FileId file_id, bool in_sticker_set, StorerT &storer, const char *source) const {
auto it = stickers_.find(file_id);
if (it == stickers_.end() || it->second == nullptr) {
return;
}
LOG_CHECK(it != stickers_.end()) << file_id << ' ' << in_sticker_set << ' ' << source;
const Sticker *sticker = it->second.get();
bool has_sticker_set_access_hash = sticker->set_id.is_valid() && !in_sticker_set;
bool has_minithumbnail = !sticker->minithumbnail.empty();
@ -41,11 +39,10 @@ void StickersManager::store_sticker(FileId file_id, bool in_sticker_set, StorerT
store(sticker->set_id.get(), storer);
if (has_sticker_set_access_hash) {
auto sticker_set = get_sticker_set(sticker->set_id);
if (sticker_set != nullptr) {
CHECK(sticker_set != nullptr);
store(sticker_set->access_hash, storer);
}
}
}
store(sticker->alt, storer);
store(sticker->dimensions, storer);
store(sticker->s_thumbnail, storer);
@ -188,9 +185,7 @@ void StickersManager::store_sticker_set(const StickerSet *sticker_set, bool with
template <class ParserT>
void StickersManager::parse_sticker_set(StickerSet *sticker_set, ParserT &parser) {
if (sticker_set == nullptr) {
return;
}
CHECK(sticker_set != nullptr);
CHECK(!sticker_set->was_loaded);
bool was_inited = sticker_set->is_inited;
bool is_installed;
@ -302,9 +297,7 @@ void StickersManager::parse_sticker_set(StickerSet *sticker_set, ParserT &parser
sticker_set->sticker_ids.push_back(sticker_id);
Sticker *sticker = get_sticker(sticker_id);
if (sticker == nullptr) {
return;
}
CHECK(sticker != nullptr);
if (sticker->set_id != sticker_set->id) {
LOG_IF(ERROR, sticker->set_id.is_valid()) << "Sticker " << sticker_id << " set_id has changed";
sticker->set_id = sticker_set->id;
@ -339,9 +332,7 @@ template <class StorerT>
void StickersManager::store_sticker_set_id(StickerSetId sticker_set_id, StorerT &storer) const {
CHECK(sticker_set_id.is_valid());
const StickerSet *sticker_set = get_sticker_set(sticker_set_id);
if (sticker_set == nullptr) {
return;
}
CHECK(sticker_set != nullptr);
store(sticker_set_id.get(), storer);
store(sticker_set->access_hash, storer);
}

58
td/telegram/Td.cpp
View File

@ -152,9 +152,6 @@ namespace td {
int VERBOSITY_NAME(td_init) = VERBOSITY_NAME(DEBUG) + 3;
int VERBOSITY_NAME(td_requests) = VERBOSITY_NAME(INFO);
int VERBOSITY_NAME(messages) = VERBOSITY_NAME(INFO);
int VERBOSITY_NAME(postponed_pts_update) = VERBOSITY_NAME(DEBUG);
int VERBOSITY_NAME(add_pending_update) = VERBOSITY_NAME(DEBUG);
void Td::ResultHandler::set_td(Td *new_td) {
CHECK(td == nullptr);
@ -3107,7 +3104,6 @@ bool Td::is_preauthentication_request(int32 id) {
case td_api::getStorageStatisticsFast::ID:
case td_api::getDatabaseStatistics::ID:
case td_api::getMemoryStatistics::ID:
case td_api::optimizeMemory::ID:
case td_api::setNetworkType::ID:
case td_api::getNetworkStatistics::ID:
case td_api::addNetworkStatistics::ID:
@ -4198,10 +4194,6 @@ void Td::init_file_manager() {
send_closure(G()->storage_manager(), &StorageManager::on_new_file, size, real_size, cnt);
}
void destroy_file_source(FileId file_id) final {
td_->file_reference_manager_->memory_cleanup(file_id);
}
void on_file_updated(FileId file_id) final {
send_closure(G()->td(), &Td::send_update,
make_tl_object<td_api::updateFile>(td_->file_manager_->get_file_object(file_id)));
@ -5052,18 +5044,6 @@ void Td::on_request(uint64 id, const td_api::getFile &request) {
send_closure(actor_id(this), &Td::send_result, id, file_manager_->get_file_object(FileId(request.file_id_, 0)));
}
void Td::on_request(uint64 id, const td_api::getChannelDifference &request) {
auto result = messages_manager_->run_get_channel_difference_request(request.channel_difference_id_);
if (result) {
send_closure(actor_id(this), &Td::send_result, id,
td_api::make_object<td_api::ok>());
} else {
send_closure(actor_id(this), &Td::send_result, id,
td_api::make_object<td_api::error>(
400, "Channel diffence identifier already executed or nonexistent"));
}
}
void Td::on_request(uint64 id, td_api::getRemoteFile &request) {
CLEAN_INPUT_STRING(request.remote_file_id_);
auto file_type = request.file_type_ == nullptr ? FileType::Temp : get_file_type(*request.file_type_);
@ -5122,18 +5102,6 @@ void Td::on_request(uint64 id, td_api::getMemoryStatistics &request) {
memory_manager_->get_memory_stats(request.full_, std::move(query_promise));
}
void Td::on_request(uint64 id, td_api::optimizeMemory &request) {
CREATE_REQUEST_PROMISE();
auto query_promise = PromiseCreator::lambda([promise = std::move(promise)](Result<Unit> result) mutable {
if (result.is_error()) {
promise.set_error(result.move_as_error());
} else {
promise.set_value(make_tl_object<td_api::ok>());
}
});
memory_manager_->clean_memory(request.full_, std::move(query_promise));
}
void Td::on_request(uint64 id, td_api::optimizeStorage &request) {
std::vector<FileType> file_types;
@ -7547,15 +7515,6 @@ void Td::on_request(uint64 id, td_api::setOption &request) {
if (set_boolean_option("disable_group_calls")) {
return;
}
if (set_integer_option("delete_file_reference_after_seconds")) {
return;
}
if (set_integer_option("delete_user_reference_after_seconds")) {
return;
}
if (set_integer_option("delete_chat_reference_after_seconds")) {
return;
}
// End custom-patches
if (set_boolean_option("disable_persistent_network_statistics")) {
return;
@ -7570,23 +7529,6 @@ void Td::on_request(uint64 id, td_api::setOption &request) {
return;
}
break;
case 'e':
if (set_boolean_option("experiment_enable_file_reference_cleanup")) {
return;
}
if (set_boolean_option("experiment_debug_file_reference_cleanup")) {
return;
}
if (set_boolean_option("experiment_enable_chat_access_hash_cleanup")) {
return;
}
if (set_boolean_option("experiment_old_postponed_pts_updates_behavior")) {
return;
}
if (set_boolean_option("enable_reactive_channel_difference")) {
return;
}
break;
case 'i':
if (set_boolean_option("ignore_background_updates")) {
return;

7
td/telegram/Td.h
View File

@ -86,9 +86,6 @@ class WebPagesManager;
extern int VERBOSITY_NAME(td_init);
extern int VERBOSITY_NAME(td_requests);
extern int VERBOSITY_NAME(messages);
extern int VERBOSITY_NAME(postponed_pts_update);
extern int VERBOSITY_NAME(add_pending_update);
// Td may start closing after explicit "close" or "destroy" query.
// Or it may start closing by itself, because authorization is lost.
@ -543,8 +540,6 @@ class Td final : public Actor {
void on_request(uint64 id, const td_api::getFile &request);
void on_request(uint64 id, const td_api::getChannelDifference &request);
void on_request(uint64 id, td_api::getRemoteFile &request);
void on_request(uint64 id, td_api::getStorageStatistics &request);
@ -555,8 +550,6 @@ class Td final : public Actor {
void on_request(uint64 id, td_api::getMemoryStatistics &request);
void on_request(uint64 id, td_api::optimizeMemory &request);
void on_request(uint64 id, td_api::optimizeStorage &request);
void on_request(uint64 id, td_api::getNetworkStatistics &request);

17
td/telegram/UserId.h
View File

@ -11,7 +11,6 @@
#include "td/utils/common.h"
#include "td/utils/StringBuilder.h"
#include <ctime>
#include <functional>
#include <type_traits>
@ -19,13 +18,13 @@ namespace td {
class UserId {
int64 id = 0;
int64 time_ = INT64_MAX;
public:
static constexpr int64 MAX_USER_ID = (1ll << 40) - 1;
UserId() = default;
explicit UserId(int64 user_id) : id(user_id) {
set_time();
}
template <class T, typename = std::enable_if_t<std::is_convertible<T, int64>::value>>
UserId(T user_id) = delete;
@ -56,18 +55,6 @@ class UserId {
return id;
}
void set_time() {
time_ = std::time(nullptr);
}
int64 get_time() const {
return time_;
}
void reset_time() {
time_ = INT64_MAX;
}
bool operator==(const UserId &other) const {
return id == other.id;
}

42
td/telegram/VideoNotesManager.cpp
View File

@ -14,7 +14,6 @@
#include "td/telegram/telegram_api.h"
#include "td/telegram/ConfigShared.h"
#include "td/telegram/Td.h"
#include "td/utils/logging.h"
#include "td/utils/misc.h"
#include "td/utils/Status.h"
@ -26,9 +25,7 @@ VideoNotesManager::VideoNotesManager(Td *td) : td_(td) {
int32 VideoNotesManager::get_video_note_duration(FileId file_id) const {
auto it = video_notes_.find(file_id);
if (it == video_notes_.end() || it->second == nullptr) {
return 0;
}
CHECK(it != video_notes_.end());
return it->second->duration;
}
@ -38,13 +35,8 @@ tl_object_ptr<td_api::videoNote> VideoNotesManager::get_video_note_object(FileId
}
auto it = video_notes_.find(file_id);
if (it == video_notes_.end()) {
return nullptr;
}
CHECK(it != video_notes_.end());
auto video_note = it->second.get();
if (video_note == nullptr) {
return nullptr;
}
return make_tl_object<td_api::videoNote>(
video_note->duration, video_note->dimensions.width, get_minithumbnail_object(video_note->minithumbnail),
get_thumbnail_object(td_->file_manager_.get(), video_note->thumbnail, PhotoFormat::Jpeg),
@ -85,37 +77,29 @@ FileId VideoNotesManager::on_get_video_note(unique_ptr<VideoNote> new_video_note
const VideoNotesManager::VideoNote *VideoNotesManager::get_video_note(FileId file_id) const {
auto video_note = video_notes_.find(file_id);
if (video_note == video_notes_.end() ||
video_note->second == nullptr ||
video_note->second->file_id != file_id) {
return make_unique<VideoNote>().get();
if (video_note == video_notes_.end()) {
return nullptr;
}
CHECK(video_note->second->file_id == file_id);
return video_note->second.get();
}
FileId VideoNotesManager::get_video_note_thumbnail_file_id(FileId file_id) const {
auto video_note = get_video_note(file_id);
if (video_note == nullptr) {
return FileId();
}
CHECK(video_note != nullptr);
return video_note->thumbnail.file_id;
}
void VideoNotesManager::delete_video_note_thumbnail(FileId file_id) {
auto &video_note = video_notes_[file_id];
if (video_note == nullptr) {
return;
}
CHECK(video_note != nullptr);
video_note->thumbnail = PhotoSize();
}
FileId VideoNotesManager::dup_video_note(FileId new_id, FileId old_id) {
const VideoNote *old_video_note = get_video_note(old_id);
if (old_video_note == nullptr) {
return FileId();
}
CHECK(old_video_note != nullptr);
auto &new_video_note = video_notes_[new_id];
CHECK(!new_video_note);
new_video_note = make_unique<VideoNote>(*old_video_note);
@ -133,7 +117,7 @@ void VideoNotesManager::merge_video_notes(FileId new_id, FileId old_id, bool can
CHECK(old_ != nullptr);
auto new_it = video_notes_.find(new_id);
if (new_it == video_notes_.end() || new_it->second == nullptr) {
if (new_it == video_notes_.end()) {
auto &old = video_notes_[old_id];
if (!can_delete_old) {
dup_video_note(new_id, old_id);
@ -219,9 +203,7 @@ tl_object_ptr<telegram_api::InputMedia> VideoNotesManager::get_input_media(
if (input_file != nullptr) {
const VideoNote *video_note = get_video_note(file_id);
if (video_note == nullptr) {
return nullptr;
}
CHECK(video_note != nullptr);
vector<tl_object_ptr<telegram_api::DocumentAttribute>> attributes;
attributes.push_back(make_tl_object<telegram_api::documentAttributeVideo>(
@ -242,10 +224,6 @@ tl_object_ptr<telegram_api::InputMedia> VideoNotesManager::get_input_media(
return nullptr;
}
void VideoNotesManager::memory_cleanup() {
video_notes_.clear();
video_notes_.rehash(0);
}
void VideoNotesManager::memory_stats(vector<string> &output) {
output.push_back("\"video_notes_\":"); output.push_back(std::to_string(video_notes_.size()));
}

2
td/telegram/VideoNotesManager.h
View File

@ -26,8 +26,6 @@ class VideoNotesManager {
public:
explicit VideoNotesManager(Td *td);
void memory_cleanup();
void memory_stats(vector<string> &output);
int32 get_video_note_duration(FileId file_id) const;

5
td/telegram/VideoNotesManager.hpp
View File

@ -20,9 +20,7 @@ namespace td {
template <class StorerT>
void VideoNotesManager::store_video_note(FileId file_id, StorerT &storer) const {
auto it = video_notes_.find(file_id);
if (it == video_notes_.end() || it->second == nullptr) {
return;
}
CHECK(it != video_notes_.end());
const VideoNote *video_note = it->second.get();
store(video_note->duration, storer);
store(video_note->dimensions, storer);
@ -50,4 +48,5 @@ FileId VideoNotesManager::parse_video_note(ParserT &parser) {
}
return on_get_video_note(std::move(video_note), false);
}
} // namespace td

39
td/telegram/VideosManager.cpp
View File

@ -25,9 +25,7 @@ VideosManager::VideosManager(Td *td) : td_(td) {
int32 VideosManager::get_video_duration(FileId file_id) const {
auto it = videos_.find(file_id);
if (it == videos_.end() || it->second == nullptr) {
return 0;
}
CHECK(it != videos_.end());
return it->second->duration;
}
@ -37,13 +35,9 @@ tl_object_ptr<td_api::video> VideosManager::get_video_object(FileId file_id) con
}
auto it = videos_.find(file_id);
if (it == videos_.end()) {
return nullptr;
}
CHECK(it != videos_.end());
auto video = it->second.get();
if (video == nullptr) {
return nullptr;
}
CHECK(video != nullptr);
auto thumbnail = video->animated_thumbnail.file_id.is_valid()
? get_thumbnail_object(td_->file_manager_.get(), video->animated_thumbnail, PhotoFormat::Mpeg4)
: get_thumbnail_object(td_->file_manager_.get(), video->thumbnail, PhotoFormat::Jpeg);
@ -112,37 +106,29 @@ FileId VideosManager::on_get_video(unique_ptr<Video> new_video, bool replace) {
const VideosManager::Video *VideosManager::get_video(FileId file_id) const {
auto video = videos_.find(file_id);
if (video == videos_.end() ||
video->second == nullptr ||
video->second->file_id != file_id) {
return make_unique<Video>().get();
if (video == videos_.end()) {
return nullptr;
}
CHECK(video->second->file_id == file_id);
return video->second.get();
}
FileId VideosManager::get_video_thumbnail_file_id(FileId file_id) const {
auto video = get_video(file_id);
if (video == nullptr) {
return FileId();
}
CHECK(video != nullptr);
return video->thumbnail.file_id;
}
FileId VideosManager::get_video_animated_thumbnail_file_id(FileId file_id) const {
auto video = get_video(file_id);
if (video == nullptr) {
return FileId();
}
CHECK(video != nullptr);
return video->animated_thumbnail.file_id;
}
void VideosManager::delete_video_thumbnail(FileId file_id) {
auto &video = videos_[file_id];
if (video == nullptr) {
return;
}
CHECK(video != nullptr);
video->thumbnail = PhotoSize();
video->animated_thumbnail = AnimationSize();
}
@ -168,7 +154,7 @@ void VideosManager::merge_videos(FileId new_id, FileId old_id, bool can_delete_o
CHECK(old_ != nullptr);
auto new_it = videos_.find(new_id);
if (new_it == videos_.end() || new_it->second == nullptr) {
if (new_it == videos_.end()) {
auto &old = videos_[old_id];
if (!can_delete_old) {
dup_video(new_id, old_id);
@ -316,10 +302,7 @@ string VideosManager::get_video_search_text(FileId file_id) const {
CHECK(video != nullptr);
return video->file_name;
}
void VideosManager::memory_cleanup() {
videos_.clear();
videos_.rehash(0);
}
void VideosManager::memory_stats(vector<string> &output) {
output.push_back("\"videos_\":"); output.push_back(std::to_string(videos_.size()));
}

2
td/telegram/VideosManager.h
View File

@ -26,8 +26,6 @@ class VideosManager {
public:
explicit VideosManager(Td *td);
void memory_cleanup();
void memory_stats(vector<string> &output);
int32 get_video_duration(FileId file_id) const;

4
td/telegram/VideosManager.hpp
View File

@ -20,9 +20,7 @@ namespace td {
template <class StorerT>
void VideosManager::store_video(FileId file_id, StorerT &storer) const {
auto it = videos_.find(file_id);
if (it == videos_.end() || it->second == nullptr) {
return;
}
CHECK(it != videos_.end());
const Video *video = it->second.get();
bool has_animated_thumbnail = video->animated_thumbnail.file_id.is_valid();
BEGIN_STORE_FLAGS();

14
td/telegram/VoiceNotesManager.cpp
View File

@ -24,9 +24,7 @@ VoiceNotesManager::VoiceNotesManager(Td *td) : td_(td) {
int32 VoiceNotesManager::get_voice_note_duration(FileId file_id) const {
auto it = voice_notes_.find(file_id);
if (it == voice_notes_.end() || it->second == nullptr) {
return 0;
}
CHECK(it != voice_notes_.end());
return it->second->duration;
}
@ -68,13 +66,11 @@ FileId VoiceNotesManager::on_get_voice_note(unique_ptr<VoiceNote> new_voice_note
const VoiceNotesManager::VoiceNote *VoiceNotesManager::get_voice_note(FileId file_id) const {
auto voice_note = voice_notes_.find(file_id);
if (voice_note == voice_notes_.end() ||
voice_note->second == nullptr ||
voice_note->second->file_id != file_id) {
return make_unique<VoiceNote>().get();
if (voice_note == voice_notes_.end()) {
return nullptr;
}
CHECK(voice_note->second->file_id == file_id);
return voice_note->second.get();
}
@ -97,7 +93,7 @@ void VoiceNotesManager::merge_voice_notes(FileId new_id, FileId old_id, bool can
CHECK(old_ != nullptr);
auto new_it = voice_notes_.find(new_id);
if (new_it == voice_notes_.end() || new_it->second == nullptr) {
if (new_it == voice_notes_.end()) {
auto &old = voice_notes_[old_id];
if (!can_delete_old) {
dup_voice_note(new_id, old_id);

4
td/telegram/VoiceNotesManager.hpp
View File

@ -18,9 +18,7 @@ namespace td {
template <class StorerT>
void VoiceNotesManager::store_voice_note(FileId file_id, StorerT &storer) const {
auto it = voice_notes_.find(file_id);
if (it == voice_notes_.end() || it->second == nullptr) {
return;
}
CHECK(it != voice_notes_.end());
const VoiceNote *voice_note = it->second.get();
store(voice_note->mime_type, storer);
store(voice_note->duration, storer);

134
td/telegram/WebPagesManager.cpp
View File

@ -408,17 +408,13 @@ WebPagesManager::WebPagesManager(Td *td, ActorShared<> parent) : td_(td), parent
void WebPagesManager::tear_down() {
parent_.reset();
// Completely clear memory when closing, to avoid memory leaks
memory_cleanup();
}
WebPagesManager::~WebPagesManager() = default;
WebPageId WebPagesManager::on_get_web_page(tl_object_ptr<telegram_api::WebPage> &&web_page_ptr,
DialogId owner_dialog_id) {
if (web_page_ptr == nullptr) {
return WebPageId();
}
CHECK(web_page_ptr != nullptr);
LOG(DEBUG) << "Got " << to_string(web_page_ptr);
switch (web_page_ptr->get_id()) {
case telegram_api::webPageEmpty::ID: {
@ -546,15 +542,9 @@ WebPageId WebPagesManager::on_get_web_page(tl_object_ptr<telegram_api::WebPage>
void WebPagesManager::update_web_page(unique_ptr<WebPage> web_page, WebPageId web_page_id, bool from_binlog,
bool from_database) {
LOG(INFO) << "Update " << web_page_id << (from_database ? " from database" : (from_binlog ? " from binlog" : ""));
if (web_page == nullptr) {
return;
}
CHECK(web_page != nullptr);
// Start custom-patches
auto find_page = web_pages_.find(web_page_id);
if (find_page == web_pages_.end()) { return; }
auto &page = find_page->second;
// End custom-patches
auto &page = web_pages_[web_page_id];
auto old_file_ids = get_web_page_file_ids(page.get());
WebPageInstantView old_instant_view;
bool is_changed = true;
@ -674,13 +664,8 @@ void WebPagesManager::on_get_web_page_instant_view_view_count(WebPageId web_page
return;
}
// Start custom-patches
auto find_web_page_val = web_pages_.find(web_page_id);
if (find_web_page_val == web_pages_.end()) { return; }
auto &web_page_val = find_web_page_val->second;
// End custom-patches
auto *instant_view = &web_page_val->instant_view;
if (instant_view->is_empty) { return; }
auto *instant_view = &web_pages_[web_page_id]->instant_view;
CHECK(!instant_view->is_empty);
if (instant_view->view_count >= view_count) {
return;
}
@ -693,11 +678,7 @@ void WebPagesManager::on_get_web_page_instant_view_view_count(WebPageId web_page
}
void WebPagesManager::on_get_web_page_by_url(const string &url, WebPageId web_page_id, bool from_database) {
// Start custom-patches
auto find_cached_web_page_id = url_to_web_page_id_.find(url);
if (find_cached_web_page_id == url_to_web_page_id_.end()) { return; }
auto &cached_web_page_id = find_cached_web_page_id->second;
// End custom-patches
auto &cached_web_page_id = url_to_web_page_id_[url];
if (!from_database && G()->parameters().use_message_db) {
if (web_page_id.is_valid()) {
if (cached_web_page_id != web_page_id) { // not already saved
@ -721,13 +702,8 @@ void WebPagesManager::register_web_page(WebPageId web_page_id, FullMessageId ful
}
LOG(INFO) << "Register " << web_page_id << " from " << full_message_id << " from " << source;
// Start custom-patches
auto find_web_page_message_ids = web_page_messages_.find(web_page_id);
if (find_web_page_message_ids == web_page_messages_.end()) { return; }
auto &web_page_message_ids = find_web_page_message_ids->second;
// End custom-patches
bool is_inserted = web_page_message_ids.insert(full_message_id).second;
if (!is_inserted) { return; }
bool is_inserted = web_page_messages_[web_page_id].insert(full_message_id).second;
LOG_CHECK(is_inserted) << source << " " << web_page_id << " " << full_message_id;
if (!td_->auth_manager_->is_bot() && !have_web_page_force(web_page_id)) {
LOG(INFO) << "Waiting for " << web_page_id << " needed in " << full_message_id;
@ -741,13 +717,8 @@ void WebPagesManager::unregister_web_page(WebPageId web_page_id, FullMessageId f
}
LOG(INFO) << "Unregister " << web_page_id << " from " << full_message_id << " from " << source;
// Start custom-patches
auto find_message_ids = web_page_messages_.find(web_page_id);
if (find_message_ids == web_page_messages_.end()) { return; }
auto &message_ids = find_message_ids->second;
// End custom-patches
auto &message_ids = web_page_messages_[web_page_id];
auto is_deleted = message_ids.erase(full_message_id) > 0;
if (!is_deleted) { return; }
LOG_CHECK(is_deleted) << source << " " << web_page_id << " " << full_message_id;
if (message_ids.empty()) {
@ -761,9 +732,7 @@ void WebPagesManager::unregister_web_page(WebPageId web_page_id, FullMessageId f
void WebPagesManager::on_get_web_page_preview_success(int64 request_id, const string &url,
tl_object_ptr<telegram_api::MessageMedia> &&message_media_ptr,
Promise<Unit> &&promise) {
if (message_media_ptr == nullptr) {
return;
}
CHECK(message_media_ptr != nullptr);
int32 constructor_id = message_media_ptr->get_id();
if (constructor_id != telegram_api::messageMediaWebPage::ID) {
if (constructor_id == telegram_api::messageMediaEmpty::ID) {
@ -778,9 +747,7 @@ void WebPagesManager::on_get_web_page_preview_success(int64 request_id, const st
}
auto message_media_web_page = move_tl_object_as<telegram_api::messageMediaWebPage>(message_media_ptr);
if (message_media_web_page->webpage_ == nullptr) {
return;
}
CHECK(message_media_web_page->webpage_ != nullptr);
auto web_page_id = on_get_web_page(std::move(message_media_web_page->webpage_), DialogId());
if (web_page_id.is_valid() && !have_web_page(web_page_id)) {
@ -794,9 +761,9 @@ void WebPagesManager::on_get_web_page_preview_success(int64 request_id, const st
void WebPagesManager::on_get_web_page_preview_success(int64 request_id, const string &url, WebPageId web_page_id,
Promise<Unit> &&promise) {
if (!(web_page_id == WebPageId() || have_web_page(web_page_id))) { return; }
CHECK(web_page_id == WebPageId() || have_web_page(web_page_id));
if (!(got_web_page_previews_.find(request_id) == got_web_page_previews_.end())) { return; }
CHECK(got_web_page_previews_.find(request_id) == got_web_page_previews_.end());
got_web_page_previews_[request_id] = web_page_id;
if (web_page_id.is_valid() && !url.empty()) {
@ -809,7 +776,7 @@ void WebPagesManager::on_get_web_page_preview_success(int64 request_id, const st
void WebPagesManager::on_get_web_page_preview_fail(int64 request_id, const string &url, Status error,
Promise<Unit> &&promise) {
LOG(INFO) << "Clean up getting of web page preview with url \"" << url << '"';
if (!error.is_error()) { return; }
CHECK(error.is_error());
promise.set_error(std::move(error));
}
@ -859,9 +826,7 @@ tl_object_ptr<td_api::webPage> WebPagesManager::get_web_page_preview_result(int6
}
auto it = got_web_page_previews_.find(request_id);
if (it == got_web_page_previews_.end()) {
return nullptr;
}
CHECK(it != got_web_page_previews_.end());
auto web_page_id = it->second;
got_web_page_previews_.erase(it);
return get_web_page_object(web_page_id);
@ -922,9 +887,7 @@ void WebPagesManager::load_web_page_instant_view(WebPageId web_page_id, bool for
LOG(INFO) << "Load " << web_page_id << " instant view, have " << previous_queries << " previous queries";
if (previous_queries == 0) {
const WebPageInstantView *web_page_instant_view = get_web_page_instant_view(web_page_id);
if (web_page_instant_view == nullptr) {
return;
}
CHECK(web_page_instant_view != nullptr);
if (G()->parameters().use_message_db && !web_page_instant_view->was_loaded_from_database) {
LOG(INFO) << "Trying to load " << web_page_id << " instant view from database";
@ -942,7 +905,7 @@ void WebPagesManager::load_web_page_instant_view(WebPageId web_page_id, bool for
void WebPagesManager::reload_web_page_instant_view(WebPageId web_page_id) {
LOG(INFO) << "Reload " << web_page_id << " instant view";
const WebPage *web_page = get_web_page(web_page_id);
if (!(web_page != nullptr && !web_page->instant_view.is_empty)) { return; }
CHECK(web_page != nullptr && !web_page->instant_view.is_empty);
auto promise = PromiseCreator::lambda([web_page_id](Result<> result) {
send_closure(G()->web_pages_manager(), &WebPagesManager::update_web_page_instant_view_load_requests, web_page_id,
@ -961,7 +924,7 @@ void WebPagesManager::on_load_web_page_instant_view_from_database(WebPageId web_
if (G()->close_flag()) {
return;
}
if (!(G()->parameters().use_message_db)) { return; }
CHECK(G()->parameters().use_message_db);
LOG(INFO) << "Successfully loaded " << web_page_id << " instant view of size " << value.size() << " from database";
// G()->td_db()->get_sqlite_pmc()->erase(get_web_page_instant_view_database_key(web_page_id), Auto());
// return;
@ -1211,9 +1174,6 @@ tl_object_ptr<td_api::webPage> WebPagesManager::get_web_page_object(WebPageId we
if (web_page == nullptr) {
return nullptr;
}
if (&web_page->url == nullptr) {
return nullptr;
}
int32 instant_view_version = [web_page] {
if (web_page->instant_view.is_empty) {
return 0;
@ -1355,7 +1315,7 @@ void WebPagesManager::on_web_page_changed(WebPageId web_page_id, bool have_web_p
for (auto full_message_id : it->second) {
full_message_ids.push_back(full_message_id);
}
if (full_message_ids.empty()) { return; }
CHECK(!full_message_ids.empty());
for (auto full_message_id : full_message_ids) {
if (!have_web_page) {
td_->messages_manager_->delete_pending_message_web_page(full_message_id);
@ -1364,14 +1324,9 @@ void WebPagesManager::on_web_page_changed(WebPageId web_page_id, bool have_web_p
}
}
if (have_web_page) {
// Start custom-patches
auto find_web_page_message_ids = web_page_messages_.find(web_page_id);
if (find_web_page_message_ids == web_page_messages_.end()) { return; }
auto &web_page_message_ids = find_web_page_message_ids->second;
// End custom-patches
if (!(web_page_message_ids.size() == full_message_ids.size())) { return; }
CHECK(web_page_messages_[web_page_id].size() == full_message_ids.size());
} else {
if (!(web_page_messages_.count(web_page_id) == 0)) { return; }
CHECK(web_page_messages_.count(web_page_id) == 0);
}
}
auto get_it = pending_get_web_pages_.find(web_page_id);
@ -1388,14 +1343,12 @@ void WebPagesManager::on_web_page_changed(WebPageId web_page_id, bool have_web_p
const WebPagesManager::WebPage *WebPagesManager::get_web_page(WebPageId web_page_id) const {
auto p = web_pages_.find(web_page_id);
if (p == web_pages_.end() ||
p->second == nullptr) {
if (p == web_pages_.end()) {
return nullptr;
}
} else {
return p->second.get();
}
}
const WebPagesManager::WebPageInstantView *WebPagesManager::get_web_page_instant_view(WebPageId web_page_id) const {
const WebPage *web_page = get_web_page(web_page_id);
@ -1446,9 +1399,7 @@ void WebPagesManager::on_pending_web_page_timeout(WebPageId web_page_id) {
void WebPagesManager::on_get_web_page_instant_view(WebPage *web_page, tl_object_ptr<telegram_api::page> &&page,
int32 hash, DialogId owner_dialog_id) {
if (page == nullptr) {
return;
}
CHECK(page != nullptr);
std::unordered_map<int64, Photo> photos;
for (auto &photo_ptr : page->photos_) {
Photo photo = get_photo(td_->file_manager_.get(), std::move(photo_ptr), owner_dialog_id);
@ -1563,9 +1514,7 @@ void WebPagesManager::save_web_page(const WebPage *web_page, WebPageId web_page_
return;
}
if (web_page == nullptr) {
return;
}
CHECK(web_page != nullptr);
if (!from_binlog) {
WebPageLogEvent log_event(web_page_id, web_page);
auto storer = get_log_event_storer(log_event);
@ -1601,9 +1550,7 @@ void WebPagesManager::on_binlog_web_page_event(BinlogEvent &&event) {
auto web_page_id = log_event.web_page_id;
LOG(INFO) << "Add " << web_page_id << " from binlog";
auto web_page = std::move(log_event.web_page_out);
if (web_page == nullptr) {
return;
}
CHECK(web_page != nullptr);
web_page->log_event_id = event.id_;
@ -1667,7 +1614,7 @@ void WebPagesManager::on_load_web_page_from_database(WebPageId web_page_id, stri
vector<Promise<Unit>> promises;
if (it != load_web_page_from_database_queries_.end()) {
promises = std::move(it->second);
if (promises.empty()) { return; }
CHECK(!promises.empty());
load_web_page_from_database_queries_.erase(it);
}
@ -1730,16 +1677,8 @@ FileSourceId WebPagesManager::get_url_file_source_id(const string &url) {
const WebPage *web_page = get_web_page(web_page_id);
if (web_page != nullptr) {
if (!web_page->file_source_id.is_valid()) {
// Start custom-patches
auto find_web_page_val = web_pages_.find(web_page_id);
if (find_web_page_val != web_pages_.end()) {
auto &web_page_val = find_web_page_val->second;
// End custom-patches
web_page_val->file_source_id =
web_pages_[web_page_id]->file_source_id =
td_->file_reference_manager_->create_web_page_file_source(web_page->url);
// Start custom-patches
}
// End custom-patches
}
return web_page->file_source_id;
}
@ -1793,20 +1732,6 @@ vector<FileId> WebPagesManager::get_web_page_file_ids(const WebPage *web_page) c
return result;
}
void WebPagesManager::memory_cleanup() {
web_pages_.clear();
web_pages_.rehash(0);
loaded_from_database_web_pages_.clear();
loaded_from_database_web_pages_.rehash(0);
web_page_messages_.clear();
web_page_messages_.rehash(0);
got_web_page_previews_.clear();
got_web_page_previews_.rehash(0);
url_to_web_page_id_.clear();
url_to_web_page_id_.rehash(0);
url_to_file_source_id_.clear();
url_to_file_source_id_.rehash(0);
}
void WebPagesManager::memory_stats(vector<string> &output) {
output.push_back("\"web_pages_\":"); output.push_back(std::to_string(web_pages_.size()));
output.push_back(",");
@ -1821,4 +1746,5 @@ void WebPagesManager::memory_stats(vector<string> &output) {
output.push_back("\"url_to_file_source_id_\":"); output.push_back(std::to_string(url_to_file_source_id_.size()));
}
} // namespace td

2
td/telegram/WebPagesManager.h
View File

@ -37,8 +37,6 @@ class WebPagesManager final : public Actor {
public:
WebPagesManager(Td *td, ActorShared<> parent);
void memory_cleanup();
void memory_stats(vector<string> &output);
WebPagesManager(const WebPagesManager &) = delete;

19
td/telegram/files/FileId.h
View File

@ -9,7 +9,6 @@
#include "td/utils/common.h"
#include "td/utils/StringBuilder.h"
#include <ctime>
#include <functional>
#include <type_traits>
@ -18,15 +17,11 @@ namespace td {
class FileId {
int32 id = 0;
int32 remote_id = 0;
int64 time_ = INT64_MAX;
public:
FileId() {
set_time();
}
FileId() = default;
FileId(int32 file_id, int32 remote_id) : id(file_id), remote_id(remote_id) {
set_time();
}
template <class T1, class T2, typename = std::enable_if_t<std::is_convertible<T1, int32>::value>,
typename = std::enable_if_t<std::is_convertible<T2, int32>::value>>
@ -43,18 +38,6 @@ class FileId {
return id;
}
void set_time() {
time_ = std::time(nullptr);
}
int64 get_time() const {
return time_;
}
void reset_time() {
time_ = INT64_MAX;
}
int32 get_remote() const {
return remote_id;
}

433
td/telegram/files/FileManager.cpp
View File

@ -136,9 +136,7 @@ FileNode &FileNodePtr::operator*() const {
FileNode *FileNodePtr::get() const {
auto res = get_unsafe();
if (res == nullptr) {
return {};
}
CHECK(res);
return res;
}
@ -147,9 +145,7 @@ FullRemoteFileLocation *FileNodePtr::get_remote() const {
}
FileNode *FileNodePtr::get_unsafe() const {
if (file_manager_ == nullptr) {
return {};
}
CHECK(file_manager_ != nullptr);
return file_manager_->get_file_node_raw(file_id_);
}
@ -802,17 +798,11 @@ void prepare_path_for_pmc(FileType file_type, string &path) {
FileManager::FileManager(unique_ptr<Context> context) : context_(std::move(context)) {
if (G()->parameters().use_file_db) {
file_db_ = G()->td_db()->get_file_db_shared();
use_standard_algorithm_ = true;
} else {
use_standard_algorithm_ = false;
}
parent_ = context_->create_reference();
// only the original tdlib algorithm requires a next_file_id() call at startup
if (use_standard_algorithm_) {
next_file_id();
next_file_node_id();
}
std::unordered_set<string> dir_paths;
for (int32 i = 0; i < MAX_FILE_TYPE; i++) {
@ -1053,8 +1043,8 @@ bool FileManager::try_fix_partial_local_location(FileNodePtr node) {
}
FileManager::FileIdInfo *FileManager::get_file_id_info(FileId file_id) {
file_id.set_time(); // update last access time of FileId
return &file_id_info_.at(file_id.get());
CHECK(static_cast<size_t>(file_id.get()) < file_id_info_.size());
return &file_id_info_[file_id.get()];
}
FileId FileManager::dup_file_id(FileId file_id) {
@ -1089,16 +1079,7 @@ void FileManager::try_forget_file_id(FileId file_id) {
bool is_removed = td::remove(file_node->file_ids_, file_id);
CHECK(is_removed);
*info = FileIdInfo();
if (use_standard_algorithm_) {
file_id_info_.erase(file_id.get());
} else {
file_id_info_.erase(file_id.get());
// Start custom-patches
file_id.reset_time(); // delete last access time of FileId
destroy_query(file_id.get());
context_->destroy_file_source(file_id);
// End custom-patches
}
empty_file_ids_.push_back(file_id.get());
}
FileId FileManager::register_empty(FileType type) {
@ -1113,13 +1094,7 @@ void FileManager::on_file_unlink(const FullLocalFileLocation &location) {
}
auto file_id = it->second;
auto file_node = get_sync_file_node(file_id);
if (use_standard_algorithm_) {
CHECK(file_node);
} else {
if (!file_node) {
return;
}
}
file_node->drop_local_location();
try_flush_node_info(file_node, "on_file_unlink");
}
@ -1249,8 +1224,8 @@ Result<FileId> FileManager::register_file(FileData &&data, FileLocationSource fi
int32 remote_key = 0;
if (file_view.has_remote_location()) {
RemoteInfo info{file_view.remote_location(), file_location_source, file_id};
RemoteInfo stored_info;
std::tie(remote_key, stored_info) = remote_location_info_.add_and_get(info);
remote_key = remote_location_info_.add(info);
auto &stored_info = remote_location_info_.get(remote_key);
if (stored_info.file_id_ == file_id) {
get_file_id_info(file_id)->pin_flag_ = true;
new_remote = true;
@ -1706,7 +1681,7 @@ Result<FileId> FileManager::merge(FileId x_file_id, FileId y_file_id, bool no_sy
}
}
file_nodes_.erase(node_ids[other_node_i]);
file_nodes_[node_ids[other_node_i]] = nullptr;
run_generate(node);
run_download(node, false);
@ -1914,25 +1889,17 @@ void FileManager::flush_to_pmc(FileNodePtr node, bool new_remote, bool new_local
}
FileNode *FileManager::get_file_node_raw(FileId file_id, FileNodeId *file_node_id) {
if (file_id.get() <= 0) {
if (file_id.get() <= 0 || file_id.get() >= static_cast<int32>(file_id_info_.size())) {
return nullptr;
}
auto find_file_id_info = file_id_info_.find(file_id.get());
if (find_file_id_info == file_id_info_.end()) {
return nullptr;
}
FileNodeId node_id = find_file_id_info->second.node_id_;
FileNodeId node_id = file_id_info_[file_id.get()].node_id_;
if (node_id == 0) {
return nullptr;
}
auto find_file_nodes = file_nodes_.find(node_id);
if (find_file_nodes == file_nodes_.end()) {
return nullptr;
}
if (file_node_id != nullptr) {
*file_node_id = node_id;
}
return find_file_nodes->second.get();
return file_nodes_[node_id].get();
}
FileNodePtr FileManager::get_sync_file_node(FileId file_id) {
@ -3093,9 +3060,8 @@ Result<FileId> FileManager::check_input_file_id(FileType type, Result<FileId> re
int32 remote_id = file_id.get_remote();
if (remote_id == 0) {
RemoteInfo info{file_view.remote_location(), FileLocationSource::FromUser, file_id};
RemoteInfo stored_info;
std::tie(remote_id, stored_info) = remote_location_info_.add_and_get(info);
if (stored_info.file_id_ == file_id) {
remote_id = remote_location_info_.add(info);
if (remote_location_info_.get(remote_id).file_id_ == file_id) {
get_file_id_info(file_id)->pin_flag_ = true;
}
}
@ -3377,17 +3343,20 @@ string FileManager::extract_file_reference(const tl_object_ptr<telegram_api::Inp
}
FileId FileManager::next_file_id() {
auto id = file_id_seqno++;
FileId res(static_cast<int32>(id), 0);
file_id_info_[id] = {};
res.set_time(); // update last access time of FileId
if (!empty_file_ids_.empty()) {
auto res = empty_file_ids_.back();
empty_file_ids_.pop_back();
return FileId{res, 0};
}
FileId res(static_cast<int32>(file_id_info_.size()), 0);
// LOG(ERROR) << "NEXT file_id " << res;
file_id_info_.push_back({});
return res;
}
FileManager::FileNodeId FileManager::next_file_node_id() {
auto id = file_node_seqno++;
auto res = static_cast<FileNodeId>(id);
file_nodes_[id] = nullptr;
FileNodeId res = static_cast<FileNodeId>(file_nodes_.size());
file_nodes_.emplace_back(nullptr);
return res;
}
@ -3397,9 +3366,7 @@ void FileManager::on_start_download(QueryId query_id) {
}
auto query = queries_container_.get(query_id);
if (query == nullptr) {
return;
}
CHECK(query != nullptr);
auto file_id = query->file_id_;
auto file_node = get_file_node(file_id);
@ -3422,9 +3389,7 @@ void FileManager::on_partial_download(QueryId query_id, const PartialLocalFileLo
}
auto query = queries_container_.get(query_id);
if (query == nullptr) {
return;
}
CHECK(query != nullptr);
auto file_id = query->file_id_;
auto file_node = get_file_node(file_id);
@ -3453,9 +3418,7 @@ void FileManager::on_hash(QueryId query_id, string hash) {
}
auto query = queries_container_.get(query_id);
if (query == nullptr) {
return;
}
CHECK(query != nullptr);
auto file_id = query->file_id_;
@ -3478,9 +3441,7 @@ void FileManager::on_partial_upload(QueryId query_id, const PartialRemoteFileLoc
}
auto query = queries_container_.get(query_id);
if (query == nullptr) {
return;
}
CHECK(query != nullptr);
auto file_id = query->file_id_;
auto file_node = get_file_node(file_id);
@ -3621,9 +3582,7 @@ void FileManager::on_partial_generate(QueryId query_id, const PartialLocalFileLo
}
auto query = queries_container_.get(query_id);
if (query == nullptr) {
return;
}
CHECK(query != nullptr);
auto file_id = query->file_id_;
auto file_node = get_file_node(file_id);
@ -3920,334 +3879,6 @@ void FileManager::hangup() {
stop();
}
void FileManager::destroy_query(int32 file_id) {
if (use_standard_algorithm_) {
return;
}
for (auto &query_id : queries_container_.ids()) {
auto query = queries_container_.get(query_id);
if (query != nullptr && file_id == query->file_id_.get()) {
on_error(query_id, Status::Error(400, "FILE_DOWNLOAD_RESTART"));
}
}
}
void FileManager::memory_cleanup() {
memory_cleanup(false);
}
void FileManager::memory_cleanup(bool full) {
if (use_standard_algorithm_) {
return;
}
if (!full) {
LOG(INFO) << "Initial registered ids: " << file_id_info_.size() << " registered nodes: " << file_nodes_.size();
}
std::unordered_set<int32> file_to_be_deleted = {};
auto file_ttl = full ? 0 : !G()->shared_config().get_option_integer("delete_file_reference_after_seconds", 30);
/* DESTROY OLD file_id_info_ */
if (full) {
file_id_info_.clear();
} else {
auto it = file_id_info_.begin();
auto time = std::time(nullptr);
while (it != file_id_info_.end()) {
auto find_node = file_nodes_.find(it->second.node_id_);
if (find_node != file_nodes_.end()) {
auto &node = find_node->second;
if (time - node->main_file_id_.get_time() > file_ttl) {
auto can_reset = node->download_priority_ == 0;
can_reset &= node->generate_download_priority_ == 0;
can_reset &= node->download_id_ == 0;
if (can_reset) {
auto file_ids_it = node->file_ids_.begin();
while (file_ids_it != node->file_ids_.end() && can_reset) {
auto find_file = file_id_info_.find(file_ids_it->get());
if (find_file != file_id_info_.end()) {
auto &file = find_file->second;
can_reset &= file.download_priority_ == 0;
can_reset &= time - file_ids_it->get_time() > file_ttl;
}
file_ids_it++;
}
}
if (can_reset) {
node->main_file_id_.reset_time(); // delete last access time of FileId
for (auto &file_id : node->file_ids_) {
file_id.reset_time(); // delete last access time of FileId
/* DESTROY ASSOCIATED QUERIES */
destroy_query(file_id.get());
/* DESTROY ASSOCIATED LATE */
file_to_be_deleted.insert(file_id.get());
}
/* DESTROY MAIN QUERY */
destroy_query(it->first);
/* DESTROY MAIN NODE */
file_nodes_.erase(it->second.node_id_);
/* DESTROY MAIN FILE LATE */
file_to_be_deleted.insert(it->first);
}
}
} else {
/* The file has a nonexistent node associated */
file_to_be_deleted.insert(it->first);
}
it++;
}
for (auto file_id : file_to_be_deleted) {
context_->destroy_file_source({file_id, 0});
file_id_info_.erase(file_id);
}
}
/* DESTROY INVALID FILES */
if (!full) {
auto it = file_id_info_.begin();
while (it != file_id_info_.end()) {
auto is_invalid = false;
if (it->second.node_id_ != 0) {
auto find_file_node = file_nodes_.find(it->second.node_id_);
if (find_file_node == file_nodes_.end() || find_file_node->second->empty) {
is_invalid = true;
destroy_query(it->first);
context_->destroy_file_source({it->first, 0});
file_nodes_.erase(it->second.node_id_);
}
} else {
is_invalid = true;
}
if (is_invalid) {
it = file_id_info_.erase(it);
} else {
it++;
}
}
}
/* DESTROY INVALID file_nodes_ */
if (full) {
file_nodes_.clear();
} else {
auto it = file_nodes_.begin();
while (it != file_nodes_.end()) {
auto is_invalid = false;
if (it->second == nullptr || it->second->empty) {
is_invalid = true;
} else {
if (it->second->main_file_id_.empty()) {
is_invalid = true;
} else {
auto find_file_id_info = file_id_info_.find(it->second->main_file_id_.get());
if (find_file_id_info != file_id_info_.end()) {
if (find_file_id_info->second.node_id_ == 0) {
for (auto &file_id : it->second->file_ids_) {
context_->destroy_file_source(file_id);
file_id_info_.erase(file_id.get());
}
file_id_info_.erase(it->second->main_file_id_.get());
is_invalid = true;
}
} else {
// todo: is it invalid or not here?
is_invalid = true;
}
}
}
if (is_invalid) {
it = file_nodes_.erase(it);
} else {
it++;
}
}
}
/* DESTROY INVALID file_hash_to_file_id_ */
if (full) {
file_hash_to_file_id_.clear();
} else {
auto it = file_hash_to_file_id_.begin();
while (it != file_hash_to_file_id_.end()) {
auto is_invalid = false;
auto find_file = file_id_info_.find(it->second.get());
if (find_file != file_id_info_.end()) {
auto &file = find_file->second;
auto find_file_node = file_nodes_.find(file.node_id_);
if (find_file_node == file_nodes_.end() || find_file_node->second->empty) {
is_invalid = true;
file_nodes_.erase(file.node_id_);
}
} else {
is_invalid = true;
}
if (is_invalid) {
it = file_hash_to_file_id_.erase(it);
} else {
it++;
}
}
}
/* DESTROY INVALID local_location_to_file_id_ */
if (full) {
local_location_to_file_id_.clear();
} else {
auto it = local_location_to_file_id_.begin();
while (it != local_location_to_file_id_.end()) {
auto is_invalid = false;
auto find_file = file_id_info_.find(it->second.get());
if (find_file != file_id_info_.end()) {
auto &file = find_file->second;
auto find_file_node = file_nodes_.find(file.node_id_);
if (find_file_node == file_nodes_.end() || find_file_node->second->empty) {
is_invalid = true;
file_nodes_.erase(file.node_id_);
}
} else {
is_invalid = true;
}
if (is_invalid) {
it = local_location_to_file_id_.erase(it);
} else {
it++;
}
}
}
/* DESTROY INVALID generate_location_to_file_id_ */
if (full) {
generate_location_to_file_id_.clear();
} else {
auto it = generate_location_to_file_id_.begin();
while (it != generate_location_to_file_id_.end()) {
auto is_invalid = false;
auto find_file = file_id_info_.find(it->second.get());
if (find_file != file_id_info_.end()) {
auto &file = find_file->second;
auto find_file_node = file_nodes_.find(file.node_id_);
if (find_file_node == file_nodes_.end() || find_file_node->second->empty) {
is_invalid = true;
file_nodes_.erase(file.node_id_);
}
} else {
is_invalid = true;
}
if (is_invalid) {
it = generate_location_to_file_id_.erase(it);
} else {
it++;
}
}
}
/* DESTROY INVALID remote_location_info_ */
if (full) {
remote_location_info_.clear();
} else {
remote_location_info_.lock_access_mutex();
std::unordered_map<int32, RemoteInfo> old_remote_info = {};
{
auto map = remote_location_info_.get_map();
auto it = map.begin();
while (it != map.end()) {
old_remote_info[it->second] = it->first;
it++;
}
}
auto it = old_remote_info.begin();
while (it != old_remote_info.end()) {
auto is_invalid = false;
auto find_file = file_id_info_.find(it->second.file_id_.get());
if (find_file != file_id_info_.end()) {
auto &file = find_file->second;
auto find_file_node = file_nodes_.find(file.node_id_);
if (find_file_node == file_nodes_.end() || find_file_node->second->empty) {
is_invalid = true;
file_nodes_.erase(file.node_id_);
}
} else {
is_invalid = true;
}
if (is_invalid) {
remote_location_info_.erase_unsafe(it->first);
}
it++;
}
remote_location_info_.unlock_access_mutex();
}
/* DESTROY NULL file_id_info_ */
if (full) {
file_id_info_.clear();
} else {
auto it = file_id_info_.begin();
while (it != file_id_info_.end()) {
auto is_invalid = false;
if (it->second.node_id_ != 0) {
auto find_file_node = file_nodes_.find(it->second.node_id_);
if (find_file_node == file_nodes_.end() || find_file_node->second->empty) {
is_invalid = true;
context_->destroy_file_source({it->first, 0});
}
} else {
is_invalid = true;
}
if (is_invalid) {
it = file_id_info_.erase(it);
} else {
it++;
}
}
}
if (full) {
file_nodes_.rehash(0);
file_hash_to_file_id_.rehash(0);
file_id_info_.rehash(0);
} else {
file_nodes_.rehash(file_nodes_.size() + 1);
file_hash_to_file_id_.rehash(file_hash_to_file_id_.size() + 1);
file_id_info_.rehash(file_id_info_.size() + 1);
}
if (!full) {
LOG(INFO) << "Final registered ids: " << file_id_info_.size() << " registered nodes: " << file_nodes_.size();
}
}
void FileManager::memory_stats(vector<string> &output) {
output.push_back("\"file_id_info_\":"); output.push_back(std::to_string(file_id_info_.size()));
output.push_back(",");
@ -4255,17 +3886,13 @@ void FileManager::memory_stats(vector<string> &output) {
output.push_back(",");
output.push_back("\"file_hash_to_file_id_\":"); output.push_back(std::to_string(file_hash_to_file_id_.size()));
output.push_back(",");
output.push_back("\"file_id_seqno\":"); output.push_back(std::to_string(file_id_seqno));
output.push_back("\"empty_file_ids_\":"); output.push_back(std::to_string(empty_file_ids_.size()));
output.push_back(",");
output.push_back("\"file_node_seqno\":"); output.push_back(std::to_string(file_node_seqno));
output.push_back("\"pmc_id_to_file_node_id_\":"); output.push_back(std::to_string(pmc_id_to_file_node_id_.size()));
}
void FileManager::tear_down() {
parent_.reset();
if (G()->memory_manager().get_actor_unsafe()->can_manage_memory()) {
// Completely clear memory when closing, to avoid memory leaks
memory_cleanup(true);
}
}
} // namespace td

26
td/telegram/files/FileManager.h
View File

@ -68,12 +68,10 @@ struct NewRemoteFileLocation {
class FileNode {
public:
FileNode() {empty = true;}
FileNode(LocalFileLocation local, NewRemoteFileLocation remote, unique_ptr<FullGenerateFileLocation> generate,
int64 size, int64 expected_size, string remote_name, string url, DialogId owner_dialog_id,
FileEncryptionKey key, FileId main_file_id, int8 main_file_id_priority)
: empty(false)
, local_(std::move(local))
: local_(std::move(local))
, remote_(std::move(remote))
, generate_(std::move(generate))
, size_(size)
@ -121,7 +119,6 @@ class FileNode {
void on_info_flushed();
string suggested_path() const;
bool empty = false;
private:
friend class FileView;
@ -154,7 +151,7 @@ class FileNode {
FileDbId pmc_id_;
std::vector<FileId> file_ids_;
FileId main_file_id_ = {};
FileId main_file_id_;
double last_successful_force_reupload_time_ = -1e10;
@ -199,8 +196,6 @@ class FileManager;
class FileNodePtr {
public:
FileNodePtr() = default;
FileNodePtr(FileManager *file_manager) : file_manager_(file_manager) {
}
FileNodePtr(FileId file_id, FileManager *file_manager) : file_id_(file_id), file_manager_(file_manager) {
}
@ -348,8 +343,6 @@ class FileView {
class FileManager final : public FileLoadManager::Callback {
public:
void memory_cleanup();
void memory_stats(vector<string> &output);
class DownloadCallback {
@ -390,8 +383,6 @@ class FileManager final : public FileLoadManager::Callback {
virtual void on_file_updated(FileId size) = 0;
virtual void destroy_file_source(FileId file_id) = 0;
virtual bool add_file_source(FileId file_id, FileSourceId file_source_id) = 0;
virtual bool remove_file_source(FileId file_id, FileSourceId file_source_id) = 0;
@ -511,10 +502,6 @@ class FileManager final : public FileLoadManager::Callback {
FileId parse_file(ParserT &parser);
private:
void destroy_query(int32 file_id);
void memory_cleanup(bool full);
Result<FileId> check_input_file_id(FileType type, Result<FileId> result, bool is_encrypted, bool allow_zero,
bool is_secure) TD_WARN_UNUSED_RESULT;
@ -565,7 +552,6 @@ class FileManager final : public FileLoadManager::Callback {
ActorShared<> parent_;
unique_ptr<Context> context_;
std::shared_ptr<FileDbInterface> file_db_;
bool use_standard_algorithm_;
FileIdInfo *get_file_id_info(FileId file_id);
@ -583,17 +569,15 @@ class FileManager final : public FileLoadManager::Callback {
};
Enumerator<RemoteInfo> remote_location_info_;
FileNodeId file_node_seqno = 1;
int32 file_id_seqno = 1;
std::unordered_map<string, FileId> file_hash_to_file_id_;
std::map<FullLocalFileLocation, FileId> local_location_to_file_id_;
std::map<FullGenerateFileLocation, FileId> generate_location_to_file_id_;
std::map<FileDbId, int32> pmc_id_to_file_node_id_;
std::unordered_map<int32, FileIdInfo> file_id_info_;
std::unordered_map<FileNodeId, unique_ptr<FileNode>> file_nodes_;
vector<FileIdInfo> file_id_info_;
vector<int32> empty_file_ids_;
vector<unique_ptr<FileNode>> file_nodes_;
ActorOwn<FileLoadManager> file_load_manager_;
ActorOwn<FileGenerateManager> file_generate_manager_;