Commit Graph

360 Commits

Author SHA1 Message Date
wxiaoguang
509d811243
Upgrade required git version to 2.0 (#19577)
* Upgrade required git version to 2.0

* update document
2022-05-02 20:30:24 +08:00
zeripath
332b2ecd21
Prevent intermittent race in attribute reader close (#19537)
There is a potential rare race possible whereby the c.running channel could
be closed twice. Looking at the code I do not see a need for this c.running
channel and therefore I think we can remove this. (I think the c.running
might have been some attempt to prevent a hang but the use of os.Pipes should
prevent that.)

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: 6543 <6543@obermui.de>
2022-04-28 11:19:33 +08:00
6543
1ebb30e41b
Pass gitRepo down to GetRawDiff, since its used for main repo and wiki (#19461)
as per https://github.com/go-gitea/gitea/pull/19449#issuecomment-1105283931

pass gitRepo down to GetRawDiff, since its used for main repo and wiki
2022-04-25 20:45:18 +02:00
zeripath
0dcc74a8a7
Prevent dangling cat-file calls (goroutine alternative) (#19454)
If an `os/exec.Command` is passed non `*os.File` as an input/output, go
will create `os.Pipe`s and wait for their closure in `cmd.Wait()`.  If
the code following this is responsible for closing `io.Pipe`s or other
handlers then on process death from context cancellation the `Wait` can
hang.

There are two possible solutions:

1. use `os.Pipe` as the input/output as `cmd.Wait` does not wait for these.
2. create a goroutine waiting on the context cancellation that will close the inputs.

This PR provides the second option - which is a simpler change that can
be more easily backported.

Closes #19448

Signed-off-by: Andrew Thornton <art27@cantab.net>
2022-04-22 17:20:04 +02:00
zeripath
a7f0ce6207
Add uploadpack.allowAnySHA1InWant to allow --filter=blob:none with older git clients (#19430)
Older git clients need uploadpack.allowAnySHA1InWant if partial cloning is allowed.

Fix #19118

Signed-off-by: Andrew Thornton <art27@cantab.net>
2022-04-20 17:52:16 +02:00
wxiaoguang
124b072f0b
Remove git.Command.Run and git.Command.RunInDir* (#19280)
Follows #19266, #8553, Close #18553, now there are only three `Run..(&RunOpts{})` functions.
 * before: `stdout, err := RunInDir(path)`
 * now: `stdout, _, err := RunStdString(&git.RunOpts{Dir:path})`
2022-04-01 10:55:30 +08:00
Peter Gardfjäll
e28cc79c92
Improve sync performance for pull-mirrors (#19125)
This addresses https://github.com/go-gitea/gitea/issues/18352

It aims to improve performance (and resource use) of the `SyncReleasesWithTags` operation for pull-mirrors.

For large repositories with many tags, `SyncReleasesWithTags` can be a costly operation (taking several minutes to complete). The reason is two-fold:
    
1. on sync, every upstream repo tag is compared (for changes) against existing local entries in the release table to ensure that they are up-to-date.
    
2. the procedure for getting _each tag_ involves a series of git operations    
    ```bash
     git show-ref --tags -- v8.2.4477
     git cat-file -t 29ab6ce9f36660cffaad3c8789e71162e5db5d2f
     git cat-file -p 29ab6ce9f36660cffaad3c8789e71162e5db5d2f
     git rev-list --count 29ab6ce9f36660cffaad3c8789e71162e5db5d2f
     ```    

     of which the `git rev-list --count` can be particularly heavy.
    
This PR optimizes performance for pull-mirrors. We utilize the fact that a pull-mirror is always identical to its upstream and rebuild the entire release table on every sync and use a batch `git for-each-ref .. refs/tags` call to retrieve all tags in one go.
    
For large mirror repos, with hundreds of annotated tags, this brings down the duration of the sync operation from several minutes to a few seconds. A few unscientific examples run on my local machine:

- https://github.com/spring-projects/spring-boot (223 tags)
  - before: `0m28,673s`
  - after: `0m2,244s`
- https://github.com/kubernetes/kubernetes (890 tags)
  - before: `8m00s`
  - after: `0m8,520s`
- https://github.com/vim/vim (13954 tags)
  - before: `14m20,383s`
  - after: `0m35,467s`

 

I added a `foreachref` package which contains a flexible way of specifying which reference fields are of interest (`git-for-each-ref(1)`) and to produce a parser for the expected output. These could be reused in other places where `for-each-ref` is used.  I'll add unit tests for those if the overall PR looks promising.
2022-03-31 14:30:40 +02:00
wxiaoguang
b877504b03
Refactor git.Command.Run*, introduce RunWithContextString and RunWithContextBytes (#19266)
This follows 
* https://github.com/go-gitea/gitea/issues/18553

Introduce `RunWithContextString` and `RunWithContextBytes` to help the refactoring. Add related unit tests. They keep the same behavior to save stderr into err.Error() as `RunInXxx` before.

Remove `RunInDirTimeoutPipeline` `RunInDirTimeoutFullPipeline` `RunInDirTimeout` `RunInDirTimeoutEnv`  `RunInDirPipeline`  `RunInDirFullPipeline`  `RunTimeout`, `RunInDirTimeoutEnvPipeline`, `RunInDirTimeoutEnvFullPipeline`, `RunInDirTimeoutEnvFullPipelineFunc`.

Then remaining `RunInDir` `RunInDirBytes` `RunInDirWithEnv` can be easily refactored in next PR with a simple search & replace:
* before: `stdout, err := RunInDir(path)`
* next: `stdout, _, err := RunWithContextString(&git.RunContext{Dir:path})`

Other changes:
1. When `timeout <= 0`, use default. Because `timeout==0` is meaningless and could cause bugs. And now many functions becomes more simple, eg: `GitGcRepos` 9 lines to 1 line. `Fsck` 6 lines to 1 line.
2. Only set defaultCommandExecutionTimeout when the option `setting.Git.Timeout.Default > 0`
2022-03-31 13:56:22 +02:00
wxiaoguang
c83168104b
Use a more general (and faster) method to sanitize URLs with credentials (#19239)
Use a more general method to sanitize URLs with credentials: Simple and intuitive / Faster /  Remove all credentials in all URLs
2022-03-31 10:25:40 +08:00
6543
3e88af898a
Make git.OpenRepository accept Context (#19260)
* OpenRepositoryCtx -> OpenRepository
* OpenRepository -> openRepositoryWithDefaultContext, only for internal usage
2022-03-30 03:13:41 +08:00
zeripath
889a8c268c
Use full output of git show-ref --tags to get tags for PushUpdateAddTag (#19235)
Strangely #19038 appears to relate to an issue whereby a tag appears to
be listed in `git show-ref --tags` but then does not appear when `git
show-ref --tags -- short_name` is called.

As a solution though I propose to stop the second call as it is
unnecessary and only likely to cause problems.

I've also noticed that the tags calls are wildly inefficient and aren't using the common cat-files - so these have been added.

I've also noticed that the git commit-graph is not being written on mirroring - so I've also added writing this to the migration which should improve mirror rendering somewhat. 

Fix #19038

Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
2022-03-29 19:12:33 +02:00
Lunny Xiao
c29fbc6d23
Hide sensitive content on admin panel progress monitor (#19218)
Sanitize urls within git process descriptions.

Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Co-authored-by: Andrew Thornton <art27@cantab.net>
2022-03-27 12:54:09 +01:00
zeripath
41b60d94db
Do not include global arguments in process manager (#19226)
The git command by default adds a number of global arguments. These are not
helpful to be displayed in the process manager and so should be skipped for
default process descriptions.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2022-03-27 10:09:56 +01:00
zeripath
2d21d2af9e
Make migrations SKIP_TLS_VERIFY apply to git too (#19132)
Make SKIP_TLS_VERIFY apply to git data migrations too through adding the `-c http.sslVerify=false` option to the git clone command.

Fix #18998

Signed-off-by: Andrew Thornton <art27@cantab.net>
2022-03-19 14:16:38 +00:00
techknowlogick
0b15a729cf
rm .sample hooks which aren't used (#19101) 2022-03-16 10:33:07 +00:00
zeripath
8ddb5490e8
Don't show context cancelled errors in attribute reader (#19006)
Fix #18997

Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
2022-03-08 08:30:14 +00:00
6543
efd10f1ab4
git backend ignore replace objects (#18979)
* git backend ignore replace objects

* comment
2022-03-02 20:13:19 +00:00
zeripath
4482f62a26
Prevent dangling GetAttribute calls (#18754)
It appears possible that there could be a hang due to unread data from the
repo-attribute command pipes. This PR simply closes these during the defer.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2022-02-14 18:03:56 +01:00
Lunny Xiao
1b1658d887
Fix isempty detection of git repository (#18746)
* Fix isempty detection of git repository

* Fix IsEmpty check
2022-02-14 00:01:23 +08:00
Martin Scholz
26718a785a
Change git.cmd to RunWithContext (#18693)
Change all `cmd...Pipeline` commands to `cmd.RunWithContext`.

#18553

Co-authored-by: Martin Scholz <martin.scholz@versasec.com>
2022-02-11 13:47:22 +01:00
zeripath
eb748f5f3c
Add apply-patch, basic revert and cherry-pick functionality (#17902)
This code adds a simple endpoint to apply patches to repositories and
branches on gitea. This is then used along with the conflicting checking
code in #18004 to provide a basic implementation of cherry-pick revert.

Now because the buttons necessary for cherry-pick and revert have 
required us to create a dropdown next to the Browse Source button
I've also implemented Create Branch and Create Tag operations.

Fix #3880 
Fix #17986 

Signed-off-by: Andrew Thornton <art27@cantab.net>
2022-02-09 20:28:55 +00:00
6543
3043eb36bf
Delete old git.NewCommand() and use it as git.NewCommandContext() (#18552) 2022-02-06 20:01:47 +01:00
zeripath
9419dd2b62
Stop logging an error when notes are not found (#18626)
This is an unnecessary logging event.

Fix #18616

Signed-off-by: Andrew Thornton <art27@cantab.net>
2022-02-06 15:11:35 +08:00
Gusted
f87d5ea9ee
Remove go 1.15 support (#18511)
- Remove support for go 1.15(go.mod already requires go 1.16).

Co-authored-by: John Olheiser <john.olheiser@gmail.com>
2022-02-01 10:46:45 +08:00
zeripath
f7b152f126
Ensure git tag tests and others create test repos in tmpdir (#18447)
* Ensure git tag tests and other create test repos in tmpdir

There are a few places where tests appear to reuse testing repos which
causes random CI failures.

This PR simply changes these tests to ensure that cloning always happens
into new temporary directories.

Fix #18444

* Change log root for integration tests to use the REPO_TEST_DIR

There is a potential race in the drone integration tests whereby test-mysql etc
will start writing to log files causing make test-check fail.

Fix #18077

Signed-off-by: Andrew Thornton <art27@cantab.net>
2022-01-29 12:41:44 +00:00
Lunny Xiao
401e5c8174
Fix broken when no commits and default branch is not master (#18422)
* Fix broken when no commits and default branch is not master

* Fix IsEmpty check

* Improve codes

* Add timeout
2022-01-28 10:51:16 +08:00
6543
80adbebbc8
Unexport git.GlobalCommandArgs (#18376)
Unexport the git.GlobalCommandArgs variable.
2022-01-25 18:15:58 +00:00
Gusted
c2e13fb763
Fix partial cloning a repo (#18373)
- Pass the Global command args into serviceRPC.
- Fixes error with partial cloning.
- Add partial clone test
- Include diff

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2022-01-23 22:19:32 +01:00
Lunny Xiao
35fdefc1ff
Always use git command but not os.Command (#18363) 2022-01-23 00:57:52 -05:00
6543
54e9ee37a7
format with gofumpt (#18184)
* gofumpt -w -l .

* gofumpt -w -l -extra .

* Add linter

* manual fix

* change make fmt
2022-01-20 18:46:10 +01:00
zeripath
5cb0c9aa0d
Propagate context and ensure git commands run in request context (#17868)
This PR continues the work in #17125 by progressively ensuring that git
commands run within the request context.

This now means that the if there is a git repo already open in the context it will be used instead of reopening it.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2022-01-19 23:26:57 +00:00
6543
ff00b8688b
Fix NPE on try to get tag reference via API (#18245)
* fix npe

* rm gitRepo from Tag
2022-01-12 20:37:46 +00:00
luzpaz
af92473920
Fix source typos (#18227)
Follow-up to #18219
2022-01-10 23:46:26 +08:00
luzpaz
8c647bf0f6
Fix various typos (#18219)
Found via `codespell -q 3 -S ./options/locale,./vendor -L ba,pullrequest,pullrequests,readby,te,unknwon`

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2022-01-10 17:32:37 +08:00
Gusted
4b3bfd7e89
Enable partial clone by default (#18195)
- Enable partial clones(which are by default disabled from git) by
default, unless configured otherwise.
- Resolves #18190
2022-01-06 06:38:38 +01:00
zeripath
ffc08c1914
Do not read or write git reference files directly (#18079)
Git will and can pack references into packfiles and therefore if you write/read the
files directly you will get false results. Instead you should use update-ref and
show-ref. To that end I have created three new functions in git/repo_commit.go that
will do this correctly.

Related #17191

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-12-23 21:44:00 +08:00
99rgosse
e0cf3d86c4
Migrated Repository will show modifications when possible (#17191)
* Read patches to get history
2021-12-23 16:32:29 +08:00
zeripath
7cc7f0ed75
TestRepository_GetTag intermittently panics due to an NPE (#18043)
There are repeated panics in tests due to TestRepository_GetTag failing
to run properly.  This happens when we attempt to reset the internal
repo for a tag which has failed to load. The problem is - the panic that
this is causing is preventing us from finding what the real error is.

This PR simply moves the failure out so we have a chance to see what
really is failing.

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
2021-12-21 11:10:16 +08:00
Gusted
ff2fd08228
Simplify parameter types (#18006)
Remove repeated type declarations in function definitions.
2021-12-20 04:41:31 +00:00
zeripath
fb5f7791ef
Prevent off-by-one error on comments on newly appended lines (#18029)
* Prevent off-by-one error on comments on newly appended lines

There was a bug in CutDiffAroundLine whereby if a file without a terminal new line
has a patch which appends lines to it and a comment is placed on one of those lines
the comment diff will be a line out of place.

This fixes CutDiffAroundLine to simply ignore the missing terminal newline - however,
we should really improve this rendering to add a marker to say that there was a
previously missing terminal newline.

Fix #17875

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-12-20 02:15:49 +00:00
zeripath
f1e85622da
Improve TestPatch to use git read-tree -m and implement git-merge-one-file functionality (#18004)
The current TestPatch conflict code uses a plain git apply which does not properly
account for 3-way merging. However, we can improve things using `git read-tree -m` to
do a three-way merge then follow the algorithm used in merge-one-file. We can also use 
`--patience` and/or `--histogram` to generate a nicer diff for applying patches too.

Fix #13679
Fix #6417

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-12-19 05:19:25 +01:00
zeripath
8354670708
Prevent hang in git cat-file if repository is not a valid repository and other fixes (#17991)
This PR contains multiple fixes. The most important of which is:

* Prevent hang in git cat-file if the repository is not a valid repository 
    
    Unfortunately it appears that if git cat-file is run in an invalid
    repository it will hang until stdin is closed. This will result in
    deadlocked /pulls pages and dangling git cat-file calls if a broken
    repository is tried to be reviewed or pulls exists for a broken
    repository.

    Fix #14734
    Fix #9271
    Fix #16113

Otherwise there are a few small other fixes included which this PR was initially intending to fix:

* Fix panic on partial compares due to missing PullRequestWorkInProgressPrefixes
* Fix links on pulls pages  due to regression from #17551 - by making most /issues routes match /pulls too - Fix #17983
* Fix links on feeds pages due to another regression from #17551 but also fix issue with syncing tags - Fix #17943
* Add missing locale entries for oauth group claims
* Prevent NPEs if ColorFormat is called on nil users, repos or teams.
2021-12-16 19:01:14 +00:00
zeripath
9e6e1dc950
Improve checkBranchName (#17901)
The current implementation of checkBranchName is highly inefficient
involving opening the repository, the listing all of the branch names
checking them individually before then using using opened repo to get
the tags.

This PR avoids this by simply walking the references from show-ref
instead of opening the repository (in the nogogit case).

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-12-08 19:08:16 +00:00
mscherer
34b5436ae1
Refactor various strings (#17784)
Fixes #16478

Co-authored-by: Gusted <williamzijl7@hotmail.com>

Co-authored-by: Gusted <williamzijl7@hotmail.com>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2021-12-02 15:28:08 +08:00
zeripath
01087e9eef
Make Requests Processes and create process hierarchy. Associate OpenRepository with context. (#17125)
This PR registers requests with the process manager and manages hierarchy within the processes.

Git repos are then associated with a context, (usually the request's context) - with sub commands using this context as their base context.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-11-30 20:06:32 +00:00
Gusted
d1f5584039
Simplify code for wrting SHA to name-rev (#17696) 2021-11-18 04:50:22 -05:00
zeripath
3c4724d70e
Add .gitattribute assisted language detection to blame, diff and render (#17590)
Use check attribute code to check the assigned language of a file and send that in to
chroma as a hint for the language of the file.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-11-17 20:37:00 +00:00
wxiaoguang
750a8465f5
A better go code formatter, and now make fmt can run in Windows (#17684)
* go build / format tools
* re-format imports
2021-11-17 20:34:35 +08:00
zeripath
bbffcc3aec
Multiple Escaping Improvements (#17551)
There are multiple places where Gitea does not properly escape URLs that it is building and there are multiple places where it builds urls when there is already a simpler function available to use this.
    
This is an extensive PR attempting to fix these issues.

1. The first commit in this PR looks through all href, src and links in the Gitea codebase and has attempted to catch all the places where there is potentially incomplete escaping.
2. Whilst doing this we will prefer to use functions that create URLs over recreating them by hand.
3. All uses of strings should be directly escaped - even if they are not currently expected to contain escaping characters. The main benefit to doing this will be that we can consider relaxing the constraints on user names and reponames in future. 
4. The next commit looks at escaping in the wiki and re-considers the urls that are used there. Using the improved escaping here wiki files containing '/'. (This implementation will currently still place all of the wiki files the root directory of the repo but this would not be difficult to change.)
5. The title generation in feeds is now properly escaped.
6. EscapePound is no longer needed - urls should be PathEscaped / QueryEscaped as necessary but then re-escaped with Escape when creating html with locales Signed-off-by: Andrew Thornton <art27@cantab.net>

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-11-16 18:18:25 +00:00
KN4CK3R
f99d50fc9f
Read expected buffer size (#17409)
* Read expected buffer size.

* Changed name.
2021-10-24 22:12:43 +01:00
Lunny Xiao
4a57c9ea17
Fix some lints (#17337)
Fix some linting problems.
2021-10-17 20:47:12 +01:00
zeripath
58cd55d353
Check for context exceeded in WalkGitLog (#17319)
There is a slight race in checking of a context deadline exceed in #16467
which leads to a 500 on the repository page.

The solution is to check the error coming back from `*LogNameStatusRepoParser.Next()`
and if it is the `ContextDeadlineExceeded` break from the loop.

Fix #17314

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-10-15 19:41:34 +01:00
zeripath
a889d0cc8c
Add buttons to allow loading of incomplete diffs (#16829)
This PR adds two buttons to the stats and the end of the diffs list to load the (some of) the remaining incomplete diff sections.

Contains #16775
    
Signed-off-by: Andrew Thornton <art27@cantab.net>


## Screenshots

### Show more button at the end of the diff
![Screenshot from 2021-09-04 11-12-37](https://user-images.githubusercontent.com/1824502/132091009-b1f6113e-2c04-4be5-8a04-b8ecea56887b.png)

### Show more button at the end of the diff stats box
![Screenshot from 2021-09-04 11-14-54](https://user-images.githubusercontent.com/1824502/132091063-86da5a6d-6628-4b82-bea9-3655cd9f40f6.png)
2021-10-15 17:05:33 +01:00
zeripath
01b9d35f1a
Disable core.protectNTFS (#17300)
core.protectNTFS protects NTFS from files which may be difficult to remove or interact
with using the win32 api, however, it also appears to prevent such files from
being entered into the git indexes - fundamentally causing breakages with PRs that
affect these files. However, deliberately setting this to false may cause security
issues due to the remain sparse checkout of files in the merge pipeline.

The only sensible option therefore is to provide an optional setting which admins
could set which would forcibly switch this off if they are affected by this issue.

Fix #17092

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-10-13 14:20:11 -04:00
a1012112796
bb39359668
Add a simple way to rename branch like gh (#15870)
- Update default branch if needed
- Update protected branch if needed
- Update all not merged pull request base branch name
- Rename git branch
- Record this rename work and auto redirect for old branch on ui

Signed-off-by: a1012112796 <1012112796@qq.com>
Co-authored-by: delvh <dev.lh@web.de>
2021-10-08 19:03:04 +02:00
zeripath
001dbf100d
Defer Last Commit Info (#16467)
One of the biggest reasons for slow repository browsing is that we wait
until last commit information has been generated for all files in the
repository.

This PR proposes deferring this generation to a new POST endpoint that
does the look up outside of the main page request.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-10-08 15:08:22 +02:00
Norwin
f48dce3176
Don't return binary file changes in raw PR diffs by default (#17158)
* return diffs without binary file content change

* ?binary=true option to restore old behaviour

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Co-authored-by: zeripath <art27@cantab.net>
2021-09-27 17:09:49 -04:00
zeripath
920608e592
Support direct comparison (git diff a..b) as well merge comparison (a...b) (#16635)
This PR changes the compare page to make the "..." in the between branches a clickable
link. This changes the comparison type from "..." to "..". Similarly it makes the
initial compare icon clickable to switch the head and base branches.

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2021-09-27 13:19:34 +01:00
Alexey 〒erentyev
7e9bd206fd
Fix bundle creation (#17079)
Signed-off-by: Alexey Terentyev <axifnx@gmail.com>

Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Gwyneth Morgan <87623694+gwymor@users.noreply.github.com>
Co-authored-by: Gwyneth Morgan <gwymor@tilde.club>
2021-09-26 00:29:25 +03:00
Eng Zer Jun
f2e7d5477f
refactor: move from io/ioutil to io and os package (#17109)
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-09-22 13:38:34 +08:00
wxiaoguang
b231d0deab
Ignore Sync errors on pipes when doing CheckAttributeReader.CheckPath, fix the hang of git cat-file (#17096)
* Ignore Sync errors on pipes when doing `CheckAttributeReader.CheckPath`

* apply env patch

* Drop the Sync and fix a number of issues with the Close function

Signed-off-by: Andrew Thornton <art27@cantab.net>

* add logs for DBIndexer and CheckPath

* Fix some more closing bugs

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Add test case for language_stats

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Update modules/indexer/stats/db.go

Co-authored-by: Lauris BH <lauris@nix.lv>

Co-authored-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: 6543 <6543@obermui.de>
2021-09-20 20:46:51 +01:00
zeripath
25533657f6
Add caller to cat-file batch calls (#17082)
Some people still appear to report unclosed cat-files. This PR simply adds the caller
to the process descriptor for the CatFileBatch and CatFileBatchCheck calls.

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-09-17 20:54:15 -04:00
Jimmy Praet
3d6cb25e31
Support unprotected file patterns (#16395)
Fixes #16381

Note that changes to unprotected files via the web editor still cannot be pushed directly to the protected branch. I could easily add such support for edits and deletes if needed. But for adding, uploading or renaming unprotected files, it is not trivial.

* Extract & Move GetAffectedFiles to modules/git
2021-09-11 16:21:17 +02:00
6543
77f604a928
Add skip and limit to git.GetTags (#16897)
* Make GetTags() api similar to GetBranches()
* Use it for Tag/Release page
2021-09-10 19:30:37 +02:00
zeripath
0faf175694
Fix missing close in WalkGitLog (#17008)
When the external context is cancelled it is possible for the
GitLogReader to not itself be Closed.

This PR does three things:

1. Instead of adding a plain defer it wraps the `g.Close` in a func as
`g` may change.
2. It adds the missing explicit g.Close - although the defer fix makes
this unnecessary.
3. It passes down the external context as the base context for the
GitLogReader meaning that the cancellation of the external context will
pass down automatically.

Fix #17007

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-09-10 17:46:12 +08:00
zeripath
248b96d8a3
Use git attributes to determine generated and vendored status for language stats and diffs (#16773)
Replaces #16262
Replaces #16250
Replaces #14833

This PR first implements a `git check-attr` pipe reader - using `git check-attr --stdin -z --cached` - taking account of the change in the output format in git 1.8.5 and creates a helper function to read a tree into a temporary index file for that pipe reader.

It then wires this in to the language stats helper and into the git diff generation.

Files which are marked generated will be folded by default.

Fixes #14786
Fixes #12653
2021-09-09 21:13:36 +01:00
6543
d21702475b
Fix git.Blob.DataAsync(): close pipe since we return a NopCloser (#16899)
* make sure headGitRepo is closed on err too

* refactor

* Fix git.Blob.DataAsync(): exec cancel since we already read all bytes (close pipe since we return a NopCloser)
2021-08-31 08:43:31 +01:00
zeripath
e891d68dd7
Prevent NPE on empty commit (#16812) 2021-08-24 19:33:23 -04:00
John Olheiser
648464b504
Add bundle download for repository (#14538)
* Add bundle download

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Fix fmt

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Fix build tags

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Download specific commit

Signed-off-by: jolheiser <john.olheiser@gmail.com>
2021-08-24 11:47:09 -05:00
Lunny Xiao
f9acad82ca
Add proxy settings and support for migration and webhook (#16704)
* Add proxy settings and support for migration and webhook

* Fix default value

* Add newline for example ini

* Add lfs proxy support

* Fix lint

* Follow @zeripath's review

* Fix git clone

* Fix test

* missgin http requests for proxy

* use empty

Co-authored-by: zeripath <art27@cantab.net>

Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: zeripath <art27@cantab.net>
2021-08-18 21:10:39 +08:00
6543
2289580bb7
[API] generalize list header (#16551)
* Add info about list endpoints to CONTRIBUTING.md

* Let all list endpoints return X-Total-Count header 

* Add TODOs for GetCombinedCommitStatusByRef

* Fix models/issue_stopwatch.go

* Rrefactor models.ListDeployKeys

* Introduce helper func and use them for SetLinkHeader related func
2021-08-12 14:43:08 +02:00
nitul1991
2d25b7d44b
Add an api endpoint to fetch git notes (#15373) (#16649)
close #15373
2021-08-11 03:01:40 +02:00
KN4CK3R
d9ef43a712
Replace list.List with slices (#16311)
* Replaced list with slice.

* Fixed usage of pointer to temporary variable.

* Replaced LIFO list with slice.

* Lint

* Removed type check.

* Removed duplicated code.

* Lint

* Fixed merge.

Co-authored-by: 6543 <6543@obermui.de>
2021-08-09 14:08:51 -04:00
nitul1991
23d438f565
Change the implementation of the go-git version of GetNote to mirror the non go-git version when passed a non-existent commit (#16658)
Fixes #16657
2021-08-09 16:24:34 +01:00
a1012112796
3705168837
Add agit flow support in gitea (#14295)
* feature: add agit flow support

ref: https://git-repo.info/en/2020/03/agit-flow-and-git-repo/

example:

```Bash
git checkout -b test
echo "test" >> README.md
git commit -m "test"
git push origin HEAD:refs/for/master -o topic=test
```

Signed-off-by: a1012112796 <1012112796@qq.com>

* fix lint

* simplify code add fix some nits

* update merge help message

* Apply suggestions from code review. Thanks @jiangxin

* add forced-update message

* fix lint

* splite writePktLine

* add refs/for/<target-branch>/<topic-branch> support also

* Add test code add fix api

* fix lint

* fix test

* skip test if git version < 2.29

* try test with git 2.30.1

* fix permission check bug

* fix some nit

* logic implify and test code update

* fix bug

* apply suggestions from code review

* prepare for merge

Signed-off-by: Andrew Thornton <art27@cantab.net>

* fix permission check bug

- test code update
- apply suggestions from code review @zeripath

Signed-off-by: a1012112796 <1012112796@qq.com>

* fix bug when target branch isn't exist

* prevent some special push and fix some nits

* fix lint

* try splite

* Apply suggestions from code review

- fix permission check
- handle user rename

* fix version negotiation

* remane

* fix template

* handle empty repo

* ui: fix  branch link under the title

* fix nits

Co-authored-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2021-07-28 17:42:56 +08:00
zeripath
97381aad5d
Make cancel from CatFileBatch and CatFileBatchCheck wait for the command to end (#16479)
Fix #16427 (again!)

* handle sharing violation error code

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: Lauris BH <lauris@nix.lv>
2021-07-20 20:23:01 +03:00
6543
b26c3b482f
Add TestPrepareWikiFileName (#16487)
* Add TestPrepareWikiFileName

* use LsTree as LsFiles is index only

* ajust other tests

Co-authored-by: Andrew Thornton <art27@cantab.net>
2021-07-20 15:16:20 +02:00
luzpaz
e0296b6a6d
Fix various documentation, user-facing, and source comment typos (#16367)
* Fix various doc, user-facing, and source comment typos

Found via `codespell -q 3 -S ./options/locale,./vendor -L ba,pullrequest,pullrequests,readby`
2021-07-08 13:38:13 +02:00
zeripath
62c278e4ab
Fix modified files list in webhooks when there is a space (#16288)
* Fix modified files list in webhooks when there is a space

There is an unfortunate bug with GetCommitFileStatus where files with
spaces are misparsed and split at the space.

There is a second bug because modern gits detect renames meaning that
this function no longer works correctly.

There is a third bug in that merge commits don't have their modified
files detected correctly.

Fix #15865


Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-07-02 21:23:37 +02:00
zeripath
9979983283
Update Go-Git to take advantage of LargeObjectThreshold (#16316)
Following the merging of https://github.com/go-git/go-git/pull/330 we
can now add a setting to avoid go-git reading and caching large objects.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-06-30 22:58:45 +02:00
KN4CK3R
7d70a6eff8
Fix webhook commits wrong hash on HEAD reset (#16283)
Use `..` instead of `...` with `rev-list`. In combination with #16282 the receiver can get the correct commit. The behaviour is now like Github.

fixes #11802
2021-06-30 19:49:06 +02:00
zeripath
add74fb368
Fix panic in recursive cache (#16298)
There is a bug with last commit cache recursive cache where the last
commit information that refers to the current tree itself will cause a
panic due to its path ("") not being included in the expected tree entry
paths.

This PR fixes this by skipping the missing entry.

Fix #16290

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-06-29 16:12:43 -04:00
zeripath
19ac575d57
Limit stdout tracelog to actual stdout (#16258)
Related #16243

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-06-26 13:47:56 +01:00
Lunny Xiao
e3c626834b
Let package git depend on setting but not opposite (#15241)
* Let package git depend on setting but not opposite

* private some package variables
2021-06-26 13:28:55 +02:00
6543
3ef23d5411
Use gitea logging module for git module (#16243)
remove log() func from gogs times and switch to proper logging

Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Andrew Thornton <art27@cantab.net>
2021-06-25 17:54:08 +01:00
KN4CK3R
4cc63e9919
Fix diff expansion is missing final line in a file (#16222)
* Fixed down offset.

* Fixed wrong line count result.
2021-06-24 17:47:46 +02:00
Lunny Xiao
b223d36195
Rework repository archive (#14723)
* Use storage to store archive files

* Fix backend lint

* Add archiver table on database

* Finish archive download

* Fix test

* Add database migrations

* Add status for archiver

* Fix lint

* Add queue

* Add doctor to check and delete old archives

* Improve archive queue

* Fix tests

* improve archive storage

* Delete repo archives

* Add missing fixture

* fix fixture

* Fix fixture

* Fix test

* Fix archiver cleaning

* Fix bug

* Add docs for repository archive storage

* remove repo-archive configuration

* Fix test

* Fix test

* Fix lint

Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-06-23 17:12:38 -04:00
Viktor Yakovchuk
08f4b3f312
Fix 500 Error with branch and tag sharing the same name #15592 (#16040)
* Fix 500 Error with branch and tag sharing the same name #15592

Fixed 500 error while create Pull request when there are more
than one sources (branch, tag) with the same name

Fix #15592

Signed-off-by: Viktor Yakovchuk <viktor@yakovchuk.net>

* fix logging

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Co-authored-by: 6543 <6543@obermui.de>
2021-06-23 17:08:26 -04:00
zeripath
196593e2e9
More efficiently parse shas for shaPostProcessor (#16101)
* More efficiently parse shas for shaPostProcessor

The shaPostProcessor currently repeatedly calls git rev-parse --verify on both backends
which is fine if there is only one thing that matches a sha - however if there are
multiple things then this becomes wildly inefficient.

This PR provides functions for both backends which are much faster to use.

Fix #16092

* Add ShaExistCache to RenderContext

Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
2021-06-21 00:39:12 +02:00
zeripath
23358bc55d
Use git log name-status in get last commit (#16059)
* Improve get last commit using git log --name-status

git log --name-status -c provides information about the diff between a
commit and its parents. Using this and adjusting the algorithm to use
the first change to a path allows for a much faster generation of commit
info.

There is a subtle change in the results generated but this will cause
the results to more closely match those from elsewhere.

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Co-authored-by: Lauris BH <lauris@nix.lv>
2021-06-21 01:00:46 +03:00
zeripath
5f4522cd1f
Fix missing discard in repo_language_stats (#16030)
Set the missing discard(1) in repo_language_stats.

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: 6543 <6543@obermui.de>
2021-06-16 18:16:47 -04:00
KN4CK3R
440039c0cc
Add push to remote mirror repository (#15157)
* Added push mirror model.

* Integrated push mirror into queue.

* Moved methods into own file.

* Added basic implementation.

* Mirror wiki too.

* Removed duplicated method.

* Get url for different remotes.

* Added migration.

* Unified remote url access.

* Add/Remove push mirror remotes.

* Prevent hangs with missing credentials.

* Moved code between files.

* Changed sanitizer interface.

* Added push mirror backend methods.

* Only update the mirror remote.

* Limit refs on push.

* Added UI part.

* Added missing table.

* Delete mirror if repository gets removed.

* Changed signature. Handle object errors.

* Added upload method.

* Added "upload" unit tests.

* Added transfer adapter unit tests.

* Send correct headers.

* Added pushing of LFS objects.

* Added more logging.

* Simpler body handling.

* Process files in batches to reduce HTTP calls.

* Added created timestamp.

* Fixed invalid column name.

* Changed name to prevent xorm auto setting.

* Remove table header im empty.

* Strip exit code from error message.

* Added docs page about mirroring.

* Fixed date.

* Fixed merge errors.

* Moved test to integrations.

* Added push mirror test.

* Added test.
2021-06-14 19:20:43 +02:00
KN4CK3R
3607f79d78
Fixed assert statements. (#16089) 2021-06-07 07:27:09 +02:00
zeripath
51775f65bc
Make commit info cancelable (#16032)
* Make modules/context.Context a context.Context

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Simplify context calls

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Set the base context for requests to the HammerContext

Signed-off-by: Andrew Thornton <art27@cantab.net>

* pass context into get-last-commit

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Make commit_info cancellable

Signed-off-by: Andrew Thornton <art27@cantab.net>

* use context as context

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: 6543 <6543@obermui.de>
2021-06-06 19:44:58 -04:00
KN4CK3R
8e262104c2
Add Image Diff for SVG files (#14867)
* Added type sniffer.

* Switched content detection from base to typesniffer.

* Added GuessContentType to Blob.

* Moved image info logic to client.
Added support for SVG images in diff.

* Restore old blocked svg behaviour.

* Added missing image formats.

* Execute image diff only when container is visible.

* add margin to spinner

* improve BIN tag on image diffs

* Default to render view.

* Show image diff on incomplete diff.

Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: Lauris BH <lauris@nix.lv>
2021-06-05 15:32:19 +03:00
Norwin
a137ee7a68
git migration: don't prompt interactively for clone credentials (#15902)
* don't prompt interactively for clone credentials

* apply GIT_TERMINAL_PROMPT=0 to all git cmds

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2021-05-17 18:59:31 +08:00
KN4CK3R
df72cf6211
Fix LFS commit finder not working (#15856)
* Create a copy of the sha bytes.


Co-authored-by: Andrew Thornton <art27@cantab.net>
2021-05-14 14:12:11 +01:00
zeripath
2a565478d1
Tagger can be empty, as can Commit and Author - tolerate this (#15835)
Unfortunately some old repositories can have tags with empty Tagger, Commit
or Author. Go-Git variants will always have empty values for these whereas
the native git variant leaves them at nil. The simplest solution is just to
always have these set to empty Signatures.

v156 migration also makes the incorrect assumption that these cannot be empty.
Therefore add some handling to this and add logging and adjust broken
logging elsewhere in this migration.

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-05-12 01:11:42 -04:00
zeripath
270aab429e
On open repository open common cat file batch and batch-check (#15667)
Use common git cat-file --batch and git cat-file --batch-check to
significantly reduce calls to git.
    
Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-05-10 03:27:03 +02:00
6543
4ea0b46d9b
Tests should use test files (#15801) 2021-05-09 17:20:33 +02:00