Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Import lua 5.4.4 and set up CI. #1

Merged
merged 3 commits into from
Mar 31, 2022
Merged

Conversation

vext01
Copy link
Contributor

@vext01 vext01 commented Mar 31, 2022

Due to the way Lua development is, the git workflow is going to be a bit convoluted.

In this repo there is no main/master branch. We will have a branch for each Lua release, and we will import the upstream Lua code from tarballs.

This sets up the lua-5.4.4 branch ready for our yk patches (next PR).

vext01 added 2 commits March 31, 2022 11:08
Root dir:
    url: https://www.lua.org/ftp/lua-5.4.4.tar.gz
    sha1: 03c27684b9d5d9783fb79a7c836ba1cdc5f309cd

`tests` dir:
    url: https://www.lua.org/tests/lua-5.4.4-tests.tar.gz
    sha256: 04d28355cd67a2299dfe5708b55a0ff221ccb1a3907a3113cc103ccc05ac6aad

Lua has no official public repo, so we are having to do it this way.

Although there is https://github.com/lua/lua, this repo doesn't contain
all of the source code. For example `luac` (a useful tool for dumping
Lua bytecode) is missing.
@vext01
Copy link
Contributor Author

vext01 commented Mar 31, 2022

bors ping

1 similar comment
@vext01
Copy link
Contributor Author

vext01 commented Mar 31, 2022

bors ping

@bors
Copy link
Contributor

bors bot commented Mar 31, 2022

pong

@vext01
Copy link
Contributor Author

vext01 commented Mar 31, 2022

[yklua is already in the buildbot config, so I hope this is good to go]

@ltratt
Copy link
Contributor

ltratt commented Mar 31, 2022

Why can't we make our master branch just be whatever the latest release is? I don't see an advantage to having multiple independent branches (and some disadvantages from doing so e.g. how will we port changes).

@vext01
Copy link
Contributor Author

vext01 commented Mar 31, 2022

What would the process be for upgrading the version? How would we merge in our differences?

@ltratt
Copy link
Contributor

ltratt commented Mar 31, 2022

There's more than one way to do it, but let me throw the question back at you: how does having multiple independent branches allow us to deal with diffs?

@vext01
Copy link
Contributor Author

vext01 commented Mar 31, 2022

This way I'm able to cherry-pick our changes onto the branch for the next release.

@ltratt
Copy link
Contributor

ltratt commented Mar 31, 2022

Honestly, I'm not sure what the right way is, so although I'm a little bit sceptical, let's go with your suggestion and see how it works out. If it's not perfect, we can easily adjust in the future. Ready for me to merge?

@vext01
Copy link
Contributor Author

vext01 commented Mar 31, 2022

Honestly, I'm not sure what the right way is

To be honest, nor am I. It baffles me that there's not a public repo that we can fork.

Ready for me to merge?

Yep

@ltratt
Copy link
Contributor

ltratt commented Mar 31, 2022

bors r+

bors bot added a commit that referenced this pull request Mar 31, 2022
1: Import lua 5.4.4 and set up CI. r=ltratt a=vext01

Due to the way Lua development is, the git workflow is going to be a bit convoluted.

In this repo there is no `main`/`master` branch. We will have a branch for each Lua release, and we will import the upstream Lua code from tarballs.

This sets up the `lua-5.4.4` branch ready for our yk patches (next PR).

Co-authored-by: Edd Barrett <[email protected]>
@bors
Copy link
Contributor

bors bot commented Mar 31, 2022

Canceled.

@vext01
Copy link
Contributor Author

vext01 commented Mar 31, 2022

This is a repo that requires PT! I was using the wrong builder name in the bors config.

OK to squash?

@ltratt
Copy link
Contributor

ltratt commented Mar 31, 2022

Please squash.

@vext01 vext01 force-pushed the import-lua-5.4.4 branch from 25a869c to bb2f95e Compare March 31, 2022 10:32
@vext01
Copy link
Contributor Author

vext01 commented Mar 31, 2022

splat.

@ltratt
Copy link
Contributor

ltratt commented Mar 31, 2022

bors r+

@bors
Copy link
Contributor

bors bot commented Mar 31, 2022

Build succeeded:

@bors bors bot merged commit 3bcc7b6 into ykjit:lua-5.4.4 Mar 31, 2022
@vext01 vext01 deleted the import-lua-5.4.4 branch March 31, 2022 10:33
bors bot added a commit that referenced this pull request Aug 25, 2023
64: Track initialised yk locations r=ltratt a=Pavel-Durov

Partially fixes (only for serial compilation)  - #62

# Changes

- Remove reallocarray in favour of calloc
- Change YkLocations to be stored as pointers
- Change lyk interface to reflect its "hooks" functionality
- Move tests to a separate test script and enable serialised all.lua test suite
- Added logging in lyk module for ease of debugging
- Updated readme

`test.sh` was tested for resiliency multiple times:
```shell
$ try_repeat -v 100 sh ./test.sh
``` 
# Issues

## Uninitialised locations memory 
`reallocarray` does not initialise memory with default zero bytes locations as `calloc`. 
Moving to `calloc` and using pointers allowed to check for NULL with default values set at initialisation time.

## Tests

When lua tests are executed one by one as single files it produces different results from when it's executed via `all.lua` test suite.

Example:

```shell

$ YKD_SERIALISE_COMPILATION=1 ../src/lua -e"_U=true" ./main.lua 
...
../src/lua: ./main.lua:343: assertion failed!
stack traceback:
        [C]: in function 'assert'
        ./main.lua:343: in main chunk
        [C]: in ?
```
When run through `all.lua` it passes:
```shell
$ YKD_SERIALISE_COMPILATION=1 ../src/lua -e"_U=true" ./all.lua 
...
***** FILE 'main.lua'*****
...
***** FILE 'gc.lua'*****
...
```
Since `all.lua` is what we base the stability of yklua, I think we should include it in the tests as is, currently only serialised compilation works.

Running `all.lua` prior to these changes results in error (`main/56c5787799b876f36babbae24e9afc025806b831`):
```
$ YKD_SERIALISE_COMPILATION=1 gdb -ex 'r' -ex 'bt' --args ../src/lua -e"_U=true" ./all.lua 
...
***** FILE 'main.lua'*****

Program received signal SIGSEGV, Segmentation fault.
core::sync::atomic::AtomicUsize::fetch_sub (self=<optimised out>) at /rustc/ef85656a10657ba5e4f7fe2931a4ca6293138d51/library/core/src/sync/atomic.rs:2575
2575    /rustc/ef85656a10657ba5e4f7fe2931a4ca6293138d51/library/core/src/sync/atomic.rs: No such file or directory.
#0  core::sync::atomic::AtomicUsize::fetch_sub (self=<optimised out>) at /rustc/ef85656a10657ba5e4f7fe2931a4ca6293138d51/library/core/src/sync/atomic.rs:2575
#1  alloc::sync::{impl#33}::drop<lock_api::mutex::Mutex<parking_lot::raw_mutex::RawMutex, ykrt::location::HotLocation>, alloc::alloc::Global> (self=0x7fffffffb410)
    at /rustc/ef85656a10657ba5e4f7fe2931a4ca6293138d51/library/alloc/src/sync.rs:2370
#2  0x00007ffff7b0d4ab in core::ptr::drop_in_place<alloc::sync::Arc<lock_api::mutex::Mutex<parking_lot::raw_mutex::RawMutex, ykrt::location::HotLocation>, alloc::alloc::Global>> () at /rustc/ef85656a10657ba5e4f7fe2931a4ca6293138d51/library/core/src/ptr/mod.rs:497
#3  0x00007ffff7b0067e in core::mem::drop<alloc::sync::Arc<lock_api::mutex::Mutex<parking_lot::raw_mutex::RawMutex, ykrt::location::HotLocation>, alloc::alloc::Global>> (_x=...) at /rustc/ef85656a10657ba5e4f7fe2931a4ca6293138d51/library/core/src/mem/mod.rs:987
#4  0x00007ffff7b121f0 in ykrt::location::{impl#1}::drop (self=0x7fffffffb468) at ykrt/src/location.rs:197
#5  0x00007ffff7affa9b in core::ptr::drop_in_place<ykrt::location::Location> () at /rustc/ef85656a10657ba5e4f7fe2931a4ca6293138d51/library/core/src/ptr/mod.rs:497
#6  0x00007ffff7aff6bd in core::mem::drop<ykrt::location::Location> (_x=...) at /rustc/ef85656a10657ba5e4f7fe2931a4ca6293138d51/library/core/src/mem/mod.rs:987
#7  0x00007ffff7b004bd in ykcapi::yk_location_drop (loc=...) at ykcapi/src/lib.rs:90
#8  0x000000000088aa0e in free_loc (f=<optimised out>, i=<optimised out>, idx=<optimised out>) at lyk.c:66
#9  0x000000000088aba5 in yk_free_locactions (f=0x928cc0) at lyk.c:76
--Type <RET> for more, q to quit, c to continue without paging--
#10 0x0000000000807738 in luaF_freeproto (L=0x916e38, f=0x928cc0) at lfunc.c:276
#11 0x0000000000809fde in freeobj (L=0x916e38, o=0x928cc0) at lgc.c:767
#12 0x0000000000817145 in sweepgen (L=0x916e38, g=<optimised out>, p=<optimised out>, limit=<optimised out>, pfirstold1=<optimised out>) at lgc.c:1106
#13 0x0000000000816713 in youngcollection (L=0x916e38, g=0x916f00) at lgc.c:1239
#14 0x000000000081557d in genstep (L=0x916e38, g=0x916f00) at lgc.c:1434
#15 0x0000000000815044 in luaC_step (L=0x916e38) at lgc.c:1686
#16 0x00000000007e4447 in lua_pushstring (L=0x916e38, s=<optimised out>) at lapi.c:553
#17 0x000000000089e60c in findloader (L=0x916e38, name=0x928df8 "tracegc") at loadlib.c:641
#18 0x000000000089de0a in ll_require (L=0x916e38) at loadlib.c:666
#19 0x00000000007fda75 in precallC (L=0x916e38, func=<optimised out>, nresults=<optimised out>, f=0x89db80 <ll_require>) at ldo.c:506
#20 0x00000000007fe016 in luaD_precall (L=0x916e38, func=0x923780, nresults=1) at ldo.c:569
#21 0x0000000000883f68 in luaV_execute (L=0x916e38, ci=<optimised out>) at lvm.c:1655
#22 0x00000000007feb3b in ccall (L=0x916e38, func=<optimised out>, nResults=<optimised out>, inc=<optimised out>) at ldo.c:609
#23 0x00000000007fec61 in luaD_callnoyield (L=0x916e38, func=0x917730, nResults=-1) at ldo.c:627
#24 0x00000000007eaf03 in f_call (L=0x916e38, ud=<optimised out>) at lapi.c:1041
#25 0x00000000007f8be7 in luaD_rawrunprotected (L=0x916e38, f=0x7eae40 <f_call>, ud=0x7ffff2487308) at ldo.c:144
#26 0x0000000000801406 in luaD_pcall (L=0x916e38, func=0x7eae40 <f_call>, u=0x7ffff2487308, old_top=<optimised out>, ef=<optimised out>) at ldo.c:926
#27 0x00000000007ea9ec in lua_pcallk (L=0x916e38, nargs=<optimised out>, nresults=<optimised out>, errfunc=<optimised out>, ctx=<optimised out>, k=<optimised out>)
    at lapi.c:1067
#28 0x00000000007dc623 in docall (L=0x916e38, narg=0, nres=-1) at lua.c:160
#29 0x00000000007dbd24 in handle_script (L=0x916e38, argv=<optimised out>) at lua.c:255
#30 0x00000000007d9fe3 in pmain (L=0x916e38) at lua.c:634
#31 0x00000000007fda75 in precallC (L=0x916e38, func=<optimised out>, nresults=<optimised out>, f=0x7d97f0 <pmain>) at ldo.c:506
#32 0x00000000007fe0c8 in luaD_precall (L=0x916e38, func=0x9176f0, nresults=1) at ldo.c:572
#33 0x00000000007fea7f in ccall (L=0x916e38, func=0x9176f0, nResults=1, inc=<optimised out>) at ldo.c:607
#34 0x00000000007fec61 in luaD_callnoyield (L=0x916e38, func=0x9176f0, nResults=1) at ldo.c:627
#35 0x00000000007eaf03 in f_call (L=0x916e38, ud=<optimised out>) at lapi.c:1041
#36 0x00000000007f8be7 in luaD_rawrunprotected (L=0x916e38, f=0x7eae40 <f_call>, ud=0x7ffff2487058) at ldo.c:144
#37 0x0000000000801406 in luaD_pcall (L=0x916e38, func=0x7eae40 <f_call>, u=0x7ffff2487058, old_top=<optimised out>, ef=<optimised out>) at ldo.c:926
#38 0x00000000007ea9ec in lua_pcallk (L=0x916e38, nargs=<optimised out>, nresults=<optimised out>, errfunc=<optimised out>, ctx=<optimised out>, k=<optimised out>)
    at lapi.c:1067
#39 0x00000000007d94b0 in main (argc=<optimised out>, argv=<optimised out>) at lua.c:660
```

That's cause we're freeing uninitialised yk locations.


Co-authored-by: Pavel Durov <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants