Check-in [c6f6825723]
Overview
Comment:Started branch to replace C11 atomics with simpler (but hopefully lock-free) implementation
Downloads: Tarball | ZIP archive | SQL archive
Timelines: family | ancestors | remove-c11-atomics
Files: files | file ages | folders
SHA1:c6f68257232a7086d58f36317f895a20107520d8
User & Date: rkeene on 2016-09-22 17:35:54
Other Links: manifest | tags
Context
2016-09-22
17:35
Started branch to replace C11 atomics with simpler (but hopefully lock-free) implementation Leaf check-in: c6f6825723 user: rkeene tags: remove-c11-atomics
2016-08-17
16:23
Post-release version increment check-in: 1233e63987 user: rkeene tags: trunk
Changes

Modified README from [8ec76224c9] to [280019e35b].

51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
..
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120

   1. Logging (CFLAGS, -DFILED_DONT_LOG=1)
	It is possible to disable ALL logging from filed.  When logging is
	completely disabled interlocks (mutexes) for the logging pointer are
	not engaged and the logging functions are not compiled at all.
	This results in a slightly smaller and faster binary

   2. Kill idle connections (CFLAGS, -DFILED_DONT_TIMEOUT=1)
        Killing idle connections relies heavily upon C11 atomics.  This
        requires a relatively new version of GCC (4.9+) or other C compiler
        that implements this aspect of C11 and so it can be disabled at
        compile time (which is the only time it makes sense).  One day an
        alternate implementation might be present that uses a mutex instead
        of atomics at which point this documentation will be updated.

   3. Debugging (CFLAGS, -DFILED_DEBUG=1)
	This is an internal option and should only be used during development.

   4. Differing HTTP semantics (CFLAGS, -DFILED_NONBLOCK_HTTP=1)
	It is possible that some HTTP clients may not process the HTTP stream
	being delivered if they cannot write to the HTTP stream itself.  This
	has not been observed yet, but it is possible.  If these semantics are
	needed (and they should not be) then they can be enabled with this
	flag at the cost of performance.

   5. Differing chroot() semantics (CFLAGS, -DFILED_FAKE_CHROOT=1)
        In some cases it is desirable to mangle paths with a path prefix
        rather than call chroot() at startup.  This is less secure and slower
        and should be generally avoided -- however it may be necessary to do.
        In these cases the executable may be compiled with the
        FILED_FAKE_CHROOT C preprocessor macro defined and instead of calling
        chroot() all HTTP requests will have the root suffix specified as the
        argument to the "-r" or "--root" option prepended to them.

   6. MIME Types (MIMETYPES)
	For single-file convenience "filed" compiles the mapping of file
	extensions (the string in the filename following its last dot ("."))
	into the executable.  This mapping comes from a file in the format of
		type1   type1_extension1 type1_extension2...
		type2   type2_extension1 type2_extension2...
		...
	However it may not be desirable to include this mapping, or it may be
................................................................................
Because "filed" relies on chroot(2) and setuid(2), log files cannot reliably
be re-opened.  If you need log rotation then a second process, which can close
and re-open log files, must be used.  Any process may be used for writing logs
but if the process does not support log rotation then it will not provide that
benefit.  For example, if you wish to write logs to syslogd(8) you can use
logger(1), such as:
	# ./filed --root /www --user nobody --log '|logger -t filed' --daemon

Troubleshooting
---------------
   1. It won't compile, something about stdatomic.h not found or _Atomic not
      a valid type.

      => This is a bug in your compiler:
            https://gcc.gnu.org/bugzilla/show_bug.cgi?id=58016

         GCC 4.7.x and 4.8.x define the macro indicating that they have C11
         support and do not define the macro that C11 requires to indicate
         that C11 atomics are not available.  They should define that macro.

         You can disable the features in "filed" that require C11 atomics by
         defining FILED_DONT_TIMEOUT in the Makefile.







<
<
<
<
<
<
<
<
|


|






|








|







 







<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
51
52
53
54
55
56
57








58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
..
91
92
93
94
95
96
97
















   1. Logging (CFLAGS, -DFILED_DONT_LOG=1)
	It is possible to disable ALL logging from filed.  When logging is
	completely disabled interlocks (mutexes) for the logging pointer are
	not engaged and the logging functions are not compiled at all.
	This results in a slightly smaller and faster binary









   2. Debugging (CFLAGS, -DFILED_DEBUG=1)
	This is an internal option and should only be used during development.

   3. Differing HTTP semantics (CFLAGS, -DFILED_NONBLOCK_HTTP=1)
	It is possible that some HTTP clients may not process the HTTP stream
	being delivered if they cannot write to the HTTP stream itself.  This
	has not been observed yet, but it is possible.  If these semantics are
	needed (and they should not be) then they can be enabled with this
	flag at the cost of performance.

   4. Differing chroot() semantics (CFLAGS, -DFILED_FAKE_CHROOT=1)
        In some cases it is desirable to mangle paths with a path prefix
        rather than call chroot() at startup.  This is less secure and slower
        and should be generally avoided -- however it may be necessary to do.
        In these cases the executable may be compiled with the
        FILED_FAKE_CHROOT C preprocessor macro defined and instead of calling
        chroot() all HTTP requests will have the root suffix specified as the
        argument to the "-r" or "--root" option prepended to them.

   5. MIME Types (MIMETYPES)
	For single-file convenience "filed" compiles the mapping of file
	extensions (the string in the filename following its last dot ("."))
	into the executable.  This mapping comes from a file in the format of
		type1   type1_extension1 type1_extension2...
		type2   type2_extension1 type2_extension2...
		...
	However it may not be desirable to include this mapping, or it may be
................................................................................
Because "filed" relies on chroot(2) and setuid(2), log files cannot reliably
be re-opened.  If you need log rotation then a second process, which can close
and re-open log files, must be used.  Any process may be used for writing logs
but if the process does not support log rotation then it will not provide that
benefit.  For example, if you wish to write logs to syslogd(8) you can use
logger(1), such as:
	# ./filed --root /www --user nobody --log '|logger -t filed' --daemon















Modified filed.c from [73ad6aa47c] to [a92c173e7c].

39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
..
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
...
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
...
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
#include <stdarg.h>
#include <fcntl.h>
#include <stdio.h>
#include <errno.h>
#include <time.h>
#include <pwd.h>

/*
 * Determine if the C compiler supports C11 atomics
 */
#if __STDC_VERSION__ >= 201112L
#  ifndef __STDC_NO_ATOMICS__
#    define FILED_FEATURE_C11_ATOMICS 1
#  endif
#endif

/*
 * If the C compiler does not support C11 atomics, disable TIMEOUT support
 * since it relies upon it
 */
#ifndef FILED_FEATURE_C11_ATOMICS
#  define FILED_DONT_TIMEOUT 1
#endif

/*
 * These headers are only required for TIMEOUT support
 */
#ifndef FILED_DONT_TIMEOUT
#include <stdatomic.h>
#include <stdbool.h>
#endif

/* Compile time constants */
#define FILED_VERSION "1.19"
#define FILED_SENDFILE_MAX 16777215
#define FILED_MAX_FAILURE_COUNT 30
#define FILED_DEFAULT_TYPE "application/octet-stream"
#define FILED_PATH_BUFFER_SIZE 1010

................................................................................
#define BIND_ADDR "::"
#define CACHE_SIZE 8209
#define LOG_FILE "-"

/* Fuzzing Test Code */
#ifdef FILED_TEST_AFL
#define FILED_DONT_LOG 1
#define FILED_DONT_TIMEOUT 1
#define pthread_create(a, x, y, z) afl_pthread_create(a, x, y, z)
#define bind(x, y, z) afl_bind(x, y, z)
#define socket(x, y, z) 8193
#define listen(x, y) 0
#define accept(x, y, z) afl_accept(x, y, z)
#define close(x) { if (strcmp(#x, "random_fd") == 0) { close(x); } else { exit(0); } }
#define fclose(x) exit(0)
................................................................................
#define filed_sockettimeout_thread_init() 0
#define filed_sockettimeout_init() 0
#define filed_sockettimeout_accept(x) /**/
#define filed_sockettimeout_processing_start(x) /**/
#define filed_sockettimeout_processing_end(x) /**/
#define filed_sockettimeout_close(x) /**/
#else
_Atomic time_t filed_sockettimeout_time;
struct {
	_Atomic time_t expiration_time;
	_Atomic pthread_t thread_id;
	bool valid;
}* filed_sockettimeout_sockstatus;
long filed_sockettimeout_sockstatus_length;
int filed_sockettimeout_devnull_fd;

static int filed_sockettimeout_sockfd_in_range(int sockfd) {
	if (sockfd < 3) {
		return(0);
................................................................................
}

static void *filed_sockettimeout_thread(void *arg) {
	time_t now, expiration_time;
	pthread_t thread_id;
	long idx;
	int count;
	bool valid;

	while (1) {
		for (count = 0; count < 10; count++) {
			usleep(30000000);

			now = time(NULL);








<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<







 







<







 







|

|
|
|







 







|







39
40
41
42
43
44
45

























46
47
48
49
50
51
52
..
56
57
58
59
60
61
62

63
64
65
66
67
68
69
...
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
...
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
#include <stdarg.h>
#include <fcntl.h>
#include <stdio.h>
#include <errno.h>
#include <time.h>
#include <pwd.h>


























/* Compile time constants */
#define FILED_VERSION "1.19"
#define FILED_SENDFILE_MAX 16777215
#define FILED_MAX_FAILURE_COUNT 30
#define FILED_DEFAULT_TYPE "application/octet-stream"
#define FILED_PATH_BUFFER_SIZE 1010

................................................................................
#define BIND_ADDR "::"
#define CACHE_SIZE 8209
#define LOG_FILE "-"

/* Fuzzing Test Code */
#ifdef FILED_TEST_AFL
#define FILED_DONT_LOG 1

#define pthread_create(a, x, y, z) afl_pthread_create(a, x, y, z)
#define bind(x, y, z) afl_bind(x, y, z)
#define socket(x, y, z) 8193
#define listen(x, y) 0
#define accept(x, y, z) afl_accept(x, y, z)
#define close(x) { if (strcmp(#x, "random_fd") == 0) { close(x); } else { exit(0); } }
#define fclose(x) exit(0)
................................................................................
#define filed_sockettimeout_thread_init() 0
#define filed_sockettimeout_init() 0
#define filed_sockettimeout_accept(x) /**/
#define filed_sockettimeout_processing_start(x) /**/
#define filed_sockettimeout_processing_end(x) /**/
#define filed_sockettimeout_close(x) /**/
#else
time_t filed_sockettimeout_time;
struct {
	time_t expiration_time;
	pthread_t thread_id;
	int valid;
}* filed_sockettimeout_sockstatus;
long filed_sockettimeout_sockstatus_length;
int filed_sockettimeout_devnull_fd;

static int filed_sockettimeout_sockfd_in_range(int sockfd) {
	if (sockfd < 3) {
		return(0);
................................................................................
}

static void *filed_sockettimeout_thread(void *arg) {
	time_t now, expiration_time;
	pthread_t thread_id;
	long idx;
	int count;
	int valid;

	while (1) {
		for (count = 0; count < 10; count++) {
			usleep(30000000);

			now = time(NULL);