Hi, I just spun up Lemmy 0.17.3 in my kubernetes cluster but I’m having trouble getting it to federate with anything.

I can curl the API endpoints for local posts which all looks good, but all searches fail. In the logs for the backend the stack trace looks like it’s failing at trying to resolve the object.

My instance is https://campfyre.nickwebster.dev (which is funny because I briefly ran a hand-made social network called Campfyre from ~2014-2016)

Edit: I am now running 0.18.0 and still have the problem with search.

Edit 2: I added a RUN update-ca-certificates step to my docker container for lemmy_server and now I can do a direct connection (i.e. https://campfyre.nickwebster.dev/c/[email protected]) although search still fails.

  • FreemanEnglish
    0•1 year ago

    1. Make sure you don’t have private instance checked. No idea what it does but I noticed in the logs you can’t have both private instance and federation checked. I’m guess private is just that, no federation, just a walled garden, possibly for non-prod and test.

    2. Leave allowed instances blank. Only use blocked to defederate ones that are undesirable.

    3. Start searching instances and communities in the search feature. This will reach out to them and start syncing posts and content.

    Also, may want to look into 0.18.0. It doesn’t use web sockets for federation anymore. Which is a lot more efficient and I’m seeing less syncing issues with things like comments.

    Also don’t know why numbers don’t show in my replied in mobile

    • @[email protected]OPEnglish
      0•1 year ago

      Thanks. I’m not private and that is blank. All searches fail, although it might be federating in the background.

      I haven’t upgraded to 0.18.0 yet because the Dockerfile I wrote uses the crates.io release to build Lemmy from source but they haven’t published 0.18.0 on crates.io yet. I can change that to git though. I’ll probably do that tomorrow and see what happens.

      • FreemanEnglish
        0•1 year ago

        How are you searching? The best way I found to load up my instance was to head to this site

        https://lemmyverse.net/

        Click the house in the top right and input your instance domain. This will reformat everything to the right search term.

        Then click the little copy link (it’s the one under the primary instance name). And paste that into searc and mash search a few time. The instance will appear in search after a few tries (heavier trafficked instances are slower to respond). Then just open the instance and subscribe. That will start syncing the community. Your federated instances will slowly grow from there.

        • @[email protected]OPEnglish
          0•1 year ago

          Yes, I’ve tried searching with the correct format. I also tried directly navigating to a community from mine via a URL, which threw this interesting error:

          LemmyError { message: Some("couldnt_find_community"), inner: Other errors which are not explicitly handled
Caused by:
    0: error sending request for url (https://lemmy.ml/.well-known/webfinger?resource=acct:cryptography@lemmy.ml): error trying to connect: error:16000069:STORE routines:ossl_store_get0_loader_int:unregistered scheme:../crypto/store/store_register.c:237:scheme=file, error:80000002:system library:file_open:reason(2):../providers/implementations/storemgmt/file_store.c:267:calling stat(/usr/lib/ssl/certs), error:16000069:STORE routines:ossl_store_get0_loader_int:unregistered scheme:../crypto/store/store_register.c:237:scheme=file, error:80000002:system library:file_open:reason(2):../providers/implementations/storemgmt/file_store.c:267:calling stat(/usr/lib/ssl/certs), error:16000069:STORE routines:ossl_store_get0_loader_int:unregistered scheme:../crypto/store/store_register.c:237:scheme=file, error:80000002:system library:file_open:reason(2):../providers/implementations/storemgmt/file_store.c:267:calling stat(/usr/lib/ssl/certs), error:16000069:STORE routines:ossl_store_get0_loader_int:unregistered scheme:../crypto/store/store_register.c:237:scheme=file, error:80000002:system library:file_open:reason(2):../providers/implementations/storemgmt/file_store.c:267:calling stat(/usr/lib/ssl/certs), error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:../ssl/statem/statem_clnt.c:1889: (unable to get local issuer certificate)
    1: error trying to connect: error:16000069:STORE routines:ossl_store_get0_loader_int:unregistered scheme:../crypto/store/store_register.c:237:scheme=file, error:80000002:system library:file_open:reason(2):../providers/implementations/storemgmt/file_store.c:267:calling stat(/usr/lib/ssl/certs), error:16000069:STORE routines:ossl_store_get0_loader_int:unregistered scheme:../crypto/store/store_register.c:237:scheme=file, error:80000002:system library:file_open:reason(2):../providers/implementations/storemgmt/file_store.c:267:calling stat(/usr/lib/ssl/certs), error:16000069:STORE routines:ossl_store_get0_loader_int:unregistered scheme:../crypto/store/store_register.c:237:scheme=file, error:80000002:system library:file_open:reason(2):../providers/implementations/storemgmt/file_store.c:267:calling stat(/usr/lib/ssl/certs), error:16000069:STORE routines:ossl_store_get0_loader_int:unregistered scheme:../crypto/store/store_register.c:237:scheme=file, error:80000002:system library:file_open:reason(2):../providers/implementations/storemgmt/file_store.c:267:calling stat(/usr/lib/ssl/certs), error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:../ssl/statem/statem_clnt.c:1889: (unable to get local issuer certificate)
    2: error:16000069:STORE routines:ossl_store_get0_loader_int:unregistered scheme:../crypto/store/store_register.c:237:scheme=file, error:80000002:system library:file_open:reason(2):../providers/implementations/storemgmt/file_store.c:267:calling stat(/usr/lib/ssl/certs), error:16000069:STORE routines:ossl_store_get0_loader_int:unregistered scheme:../crypto/store/store_register.c:237:scheme=file, error:80000002:system library:file_open:reason(2):../providers/implementations/storemgmt/file_store.c:267:calling stat(/usr/lib/ssl/certs), error:16000069:STORE routines:ossl_store_get0_loader_int:unregistered scheme:../crypto/store/store_register.c:237:scheme=file, error:80000002:system library:file_open:reason(2):../providers/implementations/storemgmt/file_store.c:267:calling stat(/usr/lib/ssl/certs), error:16000069:STORE routines:ossl_store_get0_loader_int:unregistered scheme:../crypto/store/store_register.c:237:scheme=file, error:80000002:system library:file_open:reason(2):../providers/implementations/storemgmt/file_store.c:267:calling stat(/usr/lib/ssl/certs), error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:../ssl/statem/statem_clnt.c:1889: (unable to get local issuer certificate)
    3: error:16000069:STORE routines:ossl_store_get0_loader_int:unregistered scheme:../crypto/store/store_register.c:237:scheme=file, error:80000002:system library:file_open:reason(2):../providers/implementations/storemgmt/file_store.c:267:calling stat(/usr/lib/ssl/certs), error:16000069:STORE routines:ossl_store_get0_loader_int:unregistered scheme:../crypto/store/store_register.c:237:scheme=file, error:80000002:system library:file_open:reason(2):../providers/implementations/storemgmt/file_store.c:267:calling stat(/usr/lib/ssl/certs), error:16000069:STORE routines:ossl_store_get0_loader_int:unregistered scheme:../crypto/store/store_register.c:237:scheme=file, error:80000002:system library:file_open:reason(2):../providers/implementations/storemgmt/file_store.c:267:calling stat(/usr/lib/ssl/certs), error:16000069:STORE routines:ossl_store_get0_loader_int:unregistered scheme:../crypto/store/store_register.c:237:scheme=file, error:80000002:system library:file_open:reason(2):../providers/implementations/storemgmt/file_store.c:267:calling stat(/usr/lib/ssl/certs), error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:../ssl/statem/statem_clnt.c:1889:, context: SpanTrace [{ target: "lemmy_apub::fetcher", name: "resolve_actor_identifier", file: "crates/apub/src/fetcher/mod.rs", line: 21 }, { target: "lemmy_apub::api::read_community", name: "perform", fields: "\u{1b}[3mself\u{1b}[0m\u{1b}[2m=\u{1b}[0mGetCommunity { id: None, name: Some(\"[email protected]\"), auth: Some(Sensitive) }", file: "crates/apub/src/api/read_community.rs", line: 25 }, { target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "\u{1b}[3mhttp.method\u{1b}[0m\u{1b}[2m=\u{1b}[0mGET \u{1b}[3mhttp.scheme\u{1b}[0m\u{1b}[2m=\u{1b}[0m\"http\" \u{1b}[3mhttp.host\u{1b}[0m\u{1b}[2m=\u{1b}[0mcampfyre.nickwebster.dev \u{1b}[3mhttp.target\u{1b}[0m\u{1b}[2m=\u{1b}[0m/api/v3/community \u{1b}[3motel.kind\u{1b}[0m\u{1b}[2m=\u{1b}[0m\"server\" \u{1b}[3mrequest_id\u{1b}[0m\u{1b}[2m=\u{1b}[0mfff82c47-e8f0-4a4b-a475-28a09499a8c3 \u{1b}[3mhttp.status_code\u{1b}[0m\u{1b}[2m=\u{1b}[0m400 \u{1b}[3motel.status_code\u{1b}[0m\u{1b}[2m=\u{1b}[0m\"OK\"", file: "src/root_span_builder.rs", line: 16 }] }
          • FreemanEnglish
            1•1 year ago

            That is because the community hasn’t been synced. In 0.17.4 it was just a generic 404 community not found.

            When you are searching are you leaving the filter to all or setting it to community?

            Can you post a screenshot of your filter/search query?

  • terribleplanEnglish
    0•1 year ago

    I would start by kubectl exec’ing into your pod and using nsllookup/dig/whatever you have available to check whether DNS resolution is working inside there at all. I don’t think there is an easy way to break DNS resolution via config without it being pretty clear that you’re doing so, but you could try messing with the pod spec’s dnsPolicy (I often end up using ClusterFirst) and dnsConfig.

    I may be able to help more if you post your pod spec.

    • @[email protected]OPEnglish
      0•1 year ago

      Thanks. I shelled into my container and I was able to successfully do DNS requests (and full HTTP with the outside world).

      0: error sending request for url (https://lemmy.ml/.well-known/webfinger?resource=acct:[email protected]): error trying to connect: error:16000069:STORE routines:ossl_store_get0_loader_int:unregistered scheme:../crypto/store/store_register.c:237:scheme=file, error:80000002:system library:file_open:reason(2):../providers/implementations/storemgmt/file_store.c:267:calling stat(/usr/lib/ssl/certs), error:16000069:STORE routines:ossl_store_get0_loader_int:unregistered scheme

      (and so on)

      My podspec is as follows:

      volumes:
        - name: lemmy-config
          secret:
            secretName: lemmy-config
      imagePullSecrets:
        - name: ocirsecret
      automountServiceAccountToken: false
      containers:
        - name: lemmy
          image: [...]
          imagePullPolicy: Always
          env:
            - name: LEMMY_CONFIG_LOCATION
              value: /etc/lemmy-config/lemmy.hjson
          ports:
            - containerPort: 8536
          resources: {}
          securityContext:
            capabilities:
              drop:
              - CAP_MKNOD
              - CAP_NET_RAW
              - CAP_AUDIT_WRITE
          volumeMounts:
          - mountPath: /etc/lemmy-config
            name: lemmy-config
            readOnly: true
        - name: lemmy-ui
          image: [...]
          imagePullPolicy: Always
          ports:
            - containerPort: 1234
          resources: {}
          securityContext:
            capabilities:
              drop:
              - CAP_MKNOD
              - CAP_NET_RAW
              - CAP_AUDIT_WRITE
      enableServiceLinks: true
      hostname: lemmy
      restartPolicy: Always
      • terribleplanEnglish
        2•1 year ago

        Yeah, I think you were spot on in your diagnosis of that particular error in your updated OP of needing to update-ca-certificates. As far as I can tell based on that podspec you aren’t really doing anything particularly odd that I would expect to break DNS or something at the network layer.

        Is the issue you are seeing in your logs any different now, or still the same as before the ca certs?

        • nickAEnglish
          1•1 year ago

          I’m not getting the openssl error although search is still a bit wonky. Given that I can directly subscribe to a community I think it’s okay though.